After migrating Pet Store(Sun's Web application) with JDBC calls to Google App engine with schemalass datastore entities and then re- developing the same as GWT application for Google App engine platform, here are the features and pitfalls from that perspective
Features
1.Components of Google App engine - Sandbox enviornment, DataStore & Services scales independently of each other.
2. All requests to Google App engine follow the same request-response model. Web requests/Task Queues/ Cron Jobs/XMPP Messages/Bulk-loader all are made to go via Google's front-end.
3.Server-less infrastucture
4.Supports Java Servlet(Including the features of filter, session (persistent by google), session listener, internationalization (using browser dependent locale).,JSP,JDO and JPA Specifications.
5.Internationalization is not a native feature, but can be added it by using a web application framework with internationalization features.
6.Easily integrates with Google infrastructure for authentication & authorization. And also sandboxing behavior of App engine runtime makes multiple applications to run in the same server without the behavior of one application affecting other
7.Logging jars and configurations are part of web development SDK and hence no time spent on building logger wrapper. And also Google App engine runtime provides logs of all web requests for a given application from administration console,which can help the administrator of tracking DOS attacks,bandwidth etc.
8.Separation of configuration files for each component eg. web application, indexes, tasks queues, cron job etc all have their configurations and can be configured/handled separately by team.
9.Application Caching (Memcache) support is provided as part of sdk and can be easily programmed to put/retrieve the objects from distributed cache.
10.Multiple versions of application can be tested at same moment of time and hence promotes the parallel development of different iterations for same application.
11.Easy separation of static content from application content by mentioning the same in app-config.xml & Google App engine automatically uploads the content into web server and app server respectively.
12.Google App engine replicates the data in multiple locations, and hence application support team don’t have the handle to backup and archive the data for redundancy.
13.Management tools to manage the resources used by application is being done by Google itself and report for the same can be accessed via Administration console.
14.Google Accounts Integration - App engine features integration with Google Accounts, the user account system used by Google applications such as Google Mail, Google Docs and Google Calendar.
15.Supports the concept of cron - to do batch processing jobs(but with request paremeters) & task queues to do processing aynchronously for any web request(outside the context of web reuqest).
16.The runtime environment also limits the amount of clock time, CPU use and memory a single request can take. App engine keeps these limits flexible & applies limits to those applications that use up more resources to protect shared resources from the runaway applications. But the response time for application can also determine the number of requests the application can handle dynamically.
17. Some of the open source applications are being written to synchronize the data between Google App engine datastore and relational database. For e.g., AppRocket is an open-source replication engine that synchronizes Google App engine datastore and MySQL database.
18.App engine includes a tool for uploading and downloading data via the remote API. The tool can create new datastore entities using data from a comma separated values (CSV) data file & can even create CSV files with data from the app’s datastore.
19.Google App engine provides two different interfaces - high-level & low-level for various services like datastore,memcache etc. High-level APIs make application portable but low-level APIs provides more features by the google platform.
Pitfalls-
1.Too many quotas imposed by Google App engine [bandwidth, datastore request/response size etc.] makes road towards quote driven/oriented architecture.
2.Bad entity designs can lead to index explosion or using lot of bandwidth & time while retrieving/updating/creating/deleting entity from datastore.
3.Java.io , JNI , multi-threading use is not supported currently in java runtime
4.Limited support of SQL by Java runtime as Google datastore is schema less. For eg. A query cannot use inequality filters on more than 1 property, Stored procedures/triggers are not being supported.
5.If using google infrastucture for authentication & authorization, right now it supports two roles - general & admin.
6.URL Fetch Process doesn’t support secure HTTPS communication, and hence app engine is not good use case when application needs to be communicate securely with other sites/ web services.
7.When the application creates new entities and updates existing ones using the datastore API, the call returns with success or failure after creating/updating entities along with updating every corresponding index. This makes queries very fast at the expense of the entity updates & hence affects the performance of applications.
8.With secure data connector, web applications can integrate with on-premise resources(only ones available on intranet).
9.Only content management systems build on top of App engine can be used by applications as most of the CMS typically uses sql databases.
Recommendations
1.With more insight into the Google Datastore’s indexes concept, would like to recommend that this architecture should be used by those application which needs faster access to data and not getting affected by how much data is in the system or how it is distributed across multiple servers.
2.GAE can serve traditional website content too (such as documents and images), but the environment is especially designed for real-time dynamic web applications.
3.Selection of Google App engine as platform is mainly driven by no-capital cost, pay-per-use model for resources used beyond free quota, scalability, manageability, server less on-premise infrastructure but limited by its sandbox capabilities and quotas, only web application support and on-going activities to add additional features in java runtime.
4.Ideal case for small web applications with less traffic for deploying their web applications on Google infrastructure.
Monday, January 25, 2010
Wednesday, January 13, 2010
Indexes – does it matter for Developer when using Google App engine???
Most web application need to store information during the handling of request for retrieval during a later request. By far the most populate kind of data storage system for web applications has been the relational database. But Google App engine’s database system most closely resembles an object database. The design of app engine datastore is an abstraction that allows App engine to handle the details of distributing and scaling the application.
An app engine application stores its data as one or more datastore entities. An entity has one or more properties (each of which has name and value). Each entity is of named kind, which categorizes the entity for the purpose of queries. Entities are different from rows defined in tables – firstly entities of a given kind don’t need to have same properties as other entities of same kind. Secondly an entity can have a property of the same name as another entity has, but with different type of value. Also entity can have multiple values for a single property.
Every datastore entity has a unique key that is either provided by the application or generated by App engine. The key is not a property, but an independent aspect of entity. An entity’s keys cannot be changed after the entity is created. Neither can the entity’s kind. App engine uses the entity’s kind & key to locate where the entity is stored in a large collection of servers.
An application always needs to perform query to retrieve one or more entities from the datastore & App engine also supports mentioning three different parts of query:
• Kind of entity
• One or more Filters provided on the property’s value
• One or more sort orders defined on the properties
When the query is executed, it fetches all entities of the given kind that meet all of the given conditions, sorted in the order described. But Google App engine have a limitation right now, it brings entire entity when query is executed & hence entity needs to be carefully designed to run the query in Google App engine world. So if entity is going to be used in lot of data retrieval, then design the entity such that required data should be part of the entity and additional data like BLOB can be part of other entity, which can be retrieved by the parent entity when required. This design is essential since this will add up to the applications’ quota & hence good design of entity is required to optimize the use of bandwidth.
To support faster retrieval of entities of database, Google App engine brings the concept of index. Indexes are also used in relational database world for better performance of queries and are often done by database administrators (and hence are abstracted from the developer’s point of view). But the App Engine datastore maintains an index for every query an application intends to make & when the application executes a query, the datastore fetches the results directly from the corresponding index. But since datastore uses the indexes for fetching the results, do Google App engine developer needs to understand
1. How entities are indexed by datastore?
2. Do the developers need to provide indexes configuration or it will be automatically provided by Google Datastore?
How entities are indexed by datastore?
Although this point many not be if interest to some of the developers as this is somehow feature provided by Google Datastore. But if this feature is not used appropriately (using many Multi-value Properties) in an entity, then it can lead to index explosion as there is limit imposed by Google on indexes configuration. Let’s try to explain the indexes by giving an example of Data Class Book with 3 properties – title, price & year.
public class Book {
private String name;
private Float price;
private int year;
}
For this example, keys for entities of kind Book will be generated by Google App engine, not by application. So Google App engine maintains indexes in following tables for
I. Single table maintaining the keys of the entities of a given kind.
Key
Book/1234
Book/2345
Book/3456
II. 2 Tables mapped to a single property for that kind in ascending and descending order. So in this case it will be 6 tables( 2(sort order) * no of properties)
Key Name
Book/1234 Applied Mathematics
Book/3456 Programming Java
Book/2345 SOA Cookbook
So when entities of a given kind are created/updated/deleted, then indexes in the above mentioned tables are re-arranged. So when following query is executed:
Select * from Book where price> 10.00 && price < 50.00
The datastore will first check the index table defined for the Book kind with price property with ascending order of price. Then, the datastore starts scanning the index at the first entity that meets all of the filter conditions using the query's filter values. And then the datastore continues to scan the index, returning each entity, until it finds the next entity that does not meet the filter conditions, until it reaches the end of the index, or until it has collected the maximum number of results requested by the query.
Key Price
Book/1234 8.00
Book/3456 12.99
Book/2345 23.00
Book/3466 28.00
Book/3479 34.00
Book/3660 53.00
Datastore doesn’t update the indexes when the property is not set for that given entity. And also lets the see the example of index table when there are multi-valued properties defined in an entity (In our Book entity we can assume the example of the reviewers as Multi-Value Properties):
Key Reviewers(Sorted by reviewers, not by keys)
Book/3456 James Keenan
Book/1234 Ken Beck
Book/3456 Laurel Bird
Book/1234 Surendra Reddy
Book/3475 Yale Bordy
Do the developers need to provide indexes configuration or it will be automatically provided by Google Datastore?
After explaining how the datastore indexes the entity when the entities are created or updated or deleted. In following section, will determine which all queries will be automatically indexed by Datastore and which queries requires specifying indexes configuration.
App engine datastore provides automatically indexing when following queries are being executed by application code:
1. Querying the all entities of a given kind
The example of this query is - select * from Book
The following query will use the single index table defined for storing the keys for a entity of a given kind (in our case Book).
2. Querying the all entities of a given kind and using filters on one property using equality operator with no sort order.
The example of this query is Select * from Book where price = 28.00
3. Querying the all entities of a given kind and using filters on one property using inequality operator(< , > , >=, <=) with no sort order
The same example used earlier while describing how query is executed (in previous section can be used here) : Select * from Book where price> 10.00 && price < 50.00
4. Querying the all entities of a given kind and using sort order on one property and can use no or filter on same sorted property
The example of this query will be – Select * from Book order by price desc
5. Querying the all entities of a given kind by filtering or sorting by entity key
This query can be used if application is aware of key names and then wants to retrieve that key entity later on from datastore.
The cases for equality and inequality operators are being defined separately as they are given different preferences, which will be covered with complex queries.
Till now, developer doesn’t need to concern itself with the indexes concept as datastore is able to do everything with respect to indexes for querying the entities & hence providing the layer of abstraction to developer.
But any application can’t have just simple queries like mentioned in above steps & hence application needs to provide custom indexes before deploying the same online. Following queries right now requires customer indexes:
1. Query involving the multiple sort orders defined on different properties
The example for this query will be
Select * from Book order by price desc, name desc
2. Query involving the equality filters defined on one property & any other filters on other properties.
The example for this query will be
Select * from Book where year=2009 and price < 34.00
These complex queries bring about three different principles of inequality filters defined by Google Datastore:
• If a query uses inequality filters on one property and equality filters on one or more other properties, the index must be first sorted by the properties used in equality filters, then by properties used in inequality filters.
• If query uses inequality filters on one property and sort orders on one or more of the other properties, the index must be first sorted by the property used in inequality filters, then by the other desired sort orders.
For eg. Select * From Book where price>20.00 order by name desc
So as mentioned in the above rule, the entities will be first sorted by price and name. So to make the results look obvious, it is always better to add inequality operator also in sort order to make it explicit. So the correct query will be
Select * from Book where price >20.00 order by price, name desc
• A query cannot use inequality filters on more than 1 property. This principle of inequality can be considered as limitation of Google Datastore till now.
Although Google datastore do support equality filters on more than 1 property, but this type of query makes the setup of index as flux between automatic & complex queries. Google datastore follows zigzag fashion of search of entities using equality filters on more than 1 property. So it is always advised as a good practice by Google itself to use automatic indexing for this type of query. In case the algorithm is taking lot of time to return back the response to web handler, then try to customize the indexes via configuration.
There are additional operators – IN and Not Equal (!), which are mostly used in any of the queries made to database. IN Operator is broken down into Multiple equality operators check on same property and ! Operator is broken down into > and < operator check with that filtered value for that property by datastore. So following examples will cover that:
Select * from Book where Year IN (2001, 2003, 2005) Select &* from Book where Year=2001 & Year=2003 & Year=2005
Select * from Book where year != 2004 -> Select * from Book where year>2004 & year < 2004
Using ! Operator leads to third point on inequality filer that, application can’t use ! operator on more than 1 property at a given time. Although the complex queries (defined by Datastore) are not really complex, when compared to the real world SQL’s. But that is again the limitation of datastore and they might also get complex as the days goes by.
After covering how the indexes are created & used by datastore, developer would always like to know whether they are responsible for writing indexes. Application developer can specify the customer indexes for a given kind of entity in datastore-indexes.xml, under the WEB-INF directory. This manual process of indexes can be tedious, error-prone & not liked by application developers, and hence development servers automatically generate indexes for complex queries whenever the queries are executed. To use automatic index configuration, add the attribute autoGenerate="true" to your datastore-indexes.xml file's element. Automatic index configuration is also used if app does not have a datastore-indexes.xml file.
So with more understanding of what index do and how they are configured, the next question still unanswered is whether they are also distributed like their datastore? Yes, both Entities and indexes are distributed across multiple machines, and each machine scans its own index in parallel with others. Each machine returns results to App engine as it scans its own index, and App engine delivers the final result to app, as if the all the results of query were in one large index.
So what developer/designer needs to learn from this article -
1. Design entities correctly so that it doesn’t lead to index explosion along with taking lot of bandwidths when retrieved/updated/created in datastore
2. Need to know which query needs the custom indexes and which not. If there was an entry created in configuration file for index, and that query was deleted from code, then developer/deployer needs to manually delete those indexes from xml file.
3. Need to know the laws on inequality filters and need to work with Google limitations for complex queries.
With more insight into the Google Datastore’s indexes concept, would like to recommend that this architecture should be used by those application which needs faster access to data and not getting affected by how much data is in the system or how it is distributed across multiple servers.
An app engine application stores its data as one or more datastore entities. An entity has one or more properties (each of which has name and value). Each entity is of named kind, which categorizes the entity for the purpose of queries. Entities are different from rows defined in tables – firstly entities of a given kind don’t need to have same properties as other entities of same kind. Secondly an entity can have a property of the same name as another entity has, but with different type of value. Also entity can have multiple values for a single property.
Every datastore entity has a unique key that is either provided by the application or generated by App engine. The key is not a property, but an independent aspect of entity. An entity’s keys cannot be changed after the entity is created. Neither can the entity’s kind. App engine uses the entity’s kind & key to locate where the entity is stored in a large collection of servers.
An application always needs to perform query to retrieve one or more entities from the datastore & App engine also supports mentioning three different parts of query:
• Kind of entity
• One or more Filters provided on the property’s value
• One or more sort orders defined on the properties
When the query is executed, it fetches all entities of the given kind that meet all of the given conditions, sorted in the order described. But Google App engine have a limitation right now, it brings entire entity when query is executed & hence entity needs to be carefully designed to run the query in Google App engine world. So if entity is going to be used in lot of data retrieval, then design the entity such that required data should be part of the entity and additional data like BLOB can be part of other entity, which can be retrieved by the parent entity when required. This design is essential since this will add up to the applications’ quota & hence good design of entity is required to optimize the use of bandwidth.
To support faster retrieval of entities of database, Google App engine brings the concept of index. Indexes are also used in relational database world for better performance of queries and are often done by database administrators (and hence are abstracted from the developer’s point of view). But the App Engine datastore maintains an index for every query an application intends to make & when the application executes a query, the datastore fetches the results directly from the corresponding index. But since datastore uses the indexes for fetching the results, do Google App engine developer needs to understand
1. How entities are indexed by datastore?
2. Do the developers need to provide indexes configuration or it will be automatically provided by Google Datastore?
How entities are indexed by datastore?
Although this point many not be if interest to some of the developers as this is somehow feature provided by Google Datastore. But if this feature is not used appropriately (using many Multi-value Properties) in an entity, then it can lead to index explosion as there is limit imposed by Google on indexes configuration. Let’s try to explain the indexes by giving an example of Data Class Book with 3 properties – title, price & year.
public class Book {
private String name;
private Float price;
private int year;
}
For this example, keys for entities of kind Book will be generated by Google App engine, not by application. So Google App engine maintains indexes in following tables for
I. Single table maintaining the keys of the entities of a given kind.
Key
Book/1234
Book/2345
Book/3456
II. 2 Tables mapped to a single property for that kind in ascending and descending order. So in this case it will be 6 tables( 2(sort order) * no of properties)
Key Name
Book/1234 Applied Mathematics
Book/3456 Programming Java
Book/2345 SOA Cookbook
So when entities of a given kind are created/updated/deleted, then indexes in the above mentioned tables are re-arranged. So when following query is executed:
Select * from Book where price> 10.00 && price < 50.00
The datastore will first check the index table defined for the Book kind with price property with ascending order of price. Then, the datastore starts scanning the index at the first entity that meets all of the filter conditions using the query's filter values. And then the datastore continues to scan the index, returning each entity, until it finds the next entity that does not meet the filter conditions, until it reaches the end of the index, or until it has collected the maximum number of results requested by the query.
Key Price
Book/1234 8.00
Book/3456 12.99
Book/2345 23.00
Book/3466 28.00
Book/3479 34.00
Book/3660 53.00
Datastore doesn’t update the indexes when the property is not set for that given entity. And also lets the see the example of index table when there are multi-valued properties defined in an entity (In our Book entity we can assume the example of the reviewers as Multi-Value Properties):
Key Reviewers(Sorted by reviewers, not by keys)
Book/3456 James Keenan
Book/1234 Ken Beck
Book/3456 Laurel Bird
Book/1234 Surendra Reddy
Book/3475 Yale Bordy
Do the developers need to provide indexes configuration or it will be automatically provided by Google Datastore?
After explaining how the datastore indexes the entity when the entities are created or updated or deleted. In following section, will determine which all queries will be automatically indexed by Datastore and which queries requires specifying indexes configuration.
App engine datastore provides automatically indexing when following queries are being executed by application code:
1. Querying the all entities of a given kind
The example of this query is - select * from Book
The following query will use the single index table defined for storing the keys for a entity of a given kind (in our case Book).
2. Querying the all entities of a given kind and using filters on one property using equality operator with no sort order.
The example of this query is Select * from Book where price = 28.00
3. Querying the all entities of a given kind and using filters on one property using inequality operator(< , > , >=, <=) with no sort order
The same example used earlier while describing how query is executed (in previous section can be used here) : Select * from Book where price> 10.00 && price < 50.00
4. Querying the all entities of a given kind and using sort order on one property and can use no or filter on same sorted property
The example of this query will be – Select * from Book order by price desc
5. Querying the all entities of a given kind by filtering or sorting by entity key
This query can be used if application is aware of key names and then wants to retrieve that key entity later on from datastore.
The cases for equality and inequality operators are being defined separately as they are given different preferences, which will be covered with complex queries.
Till now, developer doesn’t need to concern itself with the indexes concept as datastore is able to do everything with respect to indexes for querying the entities & hence providing the layer of abstraction to developer.
But any application can’t have just simple queries like mentioned in above steps & hence application needs to provide custom indexes before deploying the same online. Following queries right now requires customer indexes:
1. Query involving the multiple sort orders defined on different properties
The example for this query will be
Select * from Book order by price desc, name desc
2. Query involving the equality filters defined on one property & any other filters on other properties.
The example for this query will be
Select * from Book where year=2009 and price < 34.00
These complex queries bring about three different principles of inequality filters defined by Google Datastore:
• If a query uses inequality filters on one property and equality filters on one or more other properties, the index must be first sorted by the properties used in equality filters, then by properties used in inequality filters.
• If query uses inequality filters on one property and sort orders on one or more of the other properties, the index must be first sorted by the property used in inequality filters, then by the other desired sort orders.
For eg. Select * From Book where price>20.00 order by name desc
So as mentioned in the above rule, the entities will be first sorted by price and name. So to make the results look obvious, it is always better to add inequality operator also in sort order to make it explicit. So the correct query will be
Select * from Book where price >20.00 order by price, name desc
• A query cannot use inequality filters on more than 1 property. This principle of inequality can be considered as limitation of Google Datastore till now.
Although Google datastore do support equality filters on more than 1 property, but this type of query makes the setup of index as flux between automatic & complex queries. Google datastore follows zigzag fashion of search of entities using equality filters on more than 1 property. So it is always advised as a good practice by Google itself to use automatic indexing for this type of query. In case the algorithm is taking lot of time to return back the response to web handler, then try to customize the indexes via configuration.
There are additional operators – IN and Not Equal (!), which are mostly used in any of the queries made to database. IN Operator is broken down into Multiple equality operators check on same property and ! Operator is broken down into > and < operator check with that filtered value for that property by datastore. So following examples will cover that:
Select * from Book where Year IN (2001, 2003, 2005) Select &* from Book where Year=2001 & Year=2003 & Year=2005
Select * from Book where year != 2004 -> Select * from Book where year>2004 & year < 2004
Using ! Operator leads to third point on inequality filer that, application can’t use ! operator on more than 1 property at a given time. Although the complex queries (defined by Datastore) are not really complex, when compared to the real world SQL’s. But that is again the limitation of datastore and they might also get complex as the days goes by.
After covering how the indexes are created & used by datastore, developer would always like to know whether they are responsible for writing indexes. Application developer can specify the customer indexes for a given kind of entity in datastore-indexes.xml, under the WEB-INF directory. This manual process of indexes can be tedious, error-prone & not liked by application developers, and hence development servers automatically generate indexes for complex queries whenever the queries are executed. To use automatic index configuration, add the attribute autoGenerate="true" to your datastore-indexes.xml file's
So with more understanding of what index do and how they are configured, the next question still unanswered is whether they are also distributed like their datastore? Yes, both Entities and indexes are distributed across multiple machines, and each machine scans its own index in parallel with others. Each machine returns results to App engine as it scans its own index, and App engine delivers the final result to app, as if the all the results of query were in one large index.
So what developer/designer needs to learn from this article -
1. Design entities correctly so that it doesn’t lead to index explosion along with taking lot of bandwidths when retrieved/updated/created in datastore
2. Need to know which query needs the custom indexes and which not. If there was an entry created in configuration file for index, and that query was deleted from code, then developer/deployer needs to manually delete those indexes from xml file.
3. Need to know the laws on inequality filters and need to work with Google limitations for complex queries.
With more insight into the Google Datastore’s indexes concept, would like to recommend that this architecture should be used by those application which needs faster access to data and not getting affected by how much data is in the system or how it is distributed across multiple servers.
Does Google App engine provide first degree Multi-tenancy???
As per Phil Wainewright's defined definition for First Degree Mutil-tenancy in his blog on ‘Degrees of Multi-tenancy’, dated June 8th, 2009 as "First-degree Multi-tenancy is a purist model where every layer/component of the architecture is shared all the way down to the database."
Google App engine provides the runtime scalable platform for applications running in google infrastructure. The runtime environment is an abstraction above the operating system that allows App engine to manage resource allocation, computation, request handling, scaling and load distribution without the application’s involvement. So their run-time environment provides multi-tenancy at runtime environment level.
Next thing to identify is whether their datastore(not database) provides first degree multi-tenancy or not. Google App engine’s database system most closely resembles an object database(is sometime compared to BigTable). The design of app engine datastore is an abstraction that allows App engine to handle the details of distributing and scaling the application.Datastore entities are schemaless and most of the multi-tenant defintions are being defined for relational databases.
Every datastore entity has a unique key that is either provided by the application or generated by App engine. The key is not a property, but an independent aspect of entity. But Google App engine appends the app-id defined in applications configuration to the key while storing this entity in datastore(not visible to application), but stores the entities in distributed datastores. So this can't be said that entities created for a given kind by a given application will be stored in the same datastore. But while reading the entities from the datastore, the app-id is always enforced to check that entities created by one application are not shared with another application.
So in that scenario, the google app engine can be called as first degree multi-tenant platform for applications. But application build on Google App engine are not multi-tenant by default and kindly check this http://apps.gepportal.com/products-getting-started/isv for more details.Support for multi-tenant applications is in process, but for now, applications can get part of the way there by using hooks in the datastore to namespace all of the entities for a particular user/tenant.
Google App engine provides the runtime scalable platform for applications running in google infrastructure. The runtime environment is an abstraction above the operating system that allows App engine to manage resource allocation, computation, request handling, scaling and load distribution without the application’s involvement. So their run-time environment provides multi-tenancy at runtime environment level.
Next thing to identify is whether their datastore(not database) provides first degree multi-tenancy or not. Google App engine’s database system most closely resembles an object database(is sometime compared to BigTable). The design of app engine datastore is an abstraction that allows App engine to handle the details of distributing and scaling the application.Datastore entities are schemaless and most of the multi-tenant defintions are being defined for relational databases.
Every datastore entity has a unique key that is either provided by the application or generated by App engine. The key is not a property, but an independent aspect of entity. But Google App engine appends the app-id defined in applications configuration to the key while storing this entity in datastore(not visible to application), but stores the entities in distributed datastores. So this can't be said that entities created for a given kind by a given application will be stored in the same datastore. But while reading the entities from the datastore, the app-id is always enforced to check that entities created by one application are not shared with another application.
So in that scenario, the google app engine can be called as first degree multi-tenant platform for applications. But application build on Google App engine are not multi-tenant by default and kindly check this http://apps.gepportal.com/products-getting-started/isv for more details.Support for multi-tenant applications is in process, but for now, applications can get part of the way there by using hooks in the datastore to namespace all of the entities for a particular user/tenant.
Friday, January 8, 2010
Evaluating Software Quality Attributes for Applications developed using Google App Engine
Evaluating Software Quality Attributes for Applications developed using Google App Engine --------By Amit Piplani
Google App engine (GAE) is a web application hosting service. GAE can serve traditional website content too (such as documents and images), but the environment is especially designed for real-time dynamic web applications.
This article will use the FURPS model for evaluating the following software quality attributes (functional & non-functional) for applications developed using Java runtime for applications:
1. Functionality - Feature set, Capabilities, Generality, Security
2. Usability - Human factors, Aesthetics, Consistency, Documentation
3. Reliability - Frequency/severity of failure, Recoverability, Predictability, Accuracy, Mean time to failure
4. Performance - Speed, Efficiency, Resource consumption, Throughput, Response time, Scalability
5. Supportability - Testability, Extensibility, Adaptability, Maintainability, Compatibility, Configurability, Serviceability, Install ability, Localizability, Portability
1. Functionality
Feature Set - Irrespective of the platform, feature spot holds the prime spot in any software development project & primary concern for software architect/designer/developer. So any web application irrespective of the platform should be able to meet the requirements.
Capabilities & Generality - Also an architect/designer needs to check whether the system capabilities can be met by Google App engine’s core components – Sandbox environment, datastore & the services(for example, applications relying on multi-threading, using lot of network bandwidth, relational database triggers, accessing the in-network data) before choosing Google as platform. Also the Google App engine costs nothing to get started and hence the capital expenditure [CapEx] at the start of the project is negligible (considering the deployment servers only). So this process can be somehow called as server- less topology at on-premise networks.
Each App Engine resource is measured against one of two kinds of quota: a billable quota or a fixed quota. Every application gets an amount of each billable quota for free. Customer will be charged only for the resources web app actually uses, and only for the amount of resources used above the free quota thresholds. Fixed quotas are resource maximums set by App Engine to ensure the integrity of the system. These resources describe the boundaries of the architecture, and all applications are expected to run within the same limits. They ensure that another app that is consuming too many resources will not affect the performance of your app.
The quotas and limits allowed by Google app engine platform needs to be checked too before making a selection & can be found on App Engine’s Site http://code.google.com/appengine/docs/quotas.html . So using the operational expenditure (OpEx) costs are directly proportional to the amount of resources used in a given calendar day. Also only Web Apps (with limited cron functionality) can be developed on Google App engine platform.
Security –
Sandbox environment increases the security feature of web app as application code can’t access the server on which it is running in the traditional sense. An application can read its own files from the file system, but it can’t write to files & it cannot read files that belong to other applications. An application can see/access the environmental variables set by App engine, but manipulations of these variables do not necessarily persist between requests. An application cannot access the networking facilities of the server hardware. App engine features integration with Google Accounts and hence can use the authentication capabilities of Google with minimal coding/configuration. Also If application is using the authentication via Google Accounts, App Engine includes HTTP headers in the response that give the information about the resources used by a given request.
So sandbox environment can act as "Blessing in disguise" with providing secured wrapper to web application but limiting the capabilities of an application & needs to be carefully checked to see if it really meets the needs of the required web application.
2. Usability
Although this concept is somehow holds the same weight age whether web application is developed using Google App engine Platform or not.
By using Google Web Toolkit (GWT) by far the quickest way to deploy a better UI experience full stack web application using the absolute best possible technologies available (Google’s scalable infrastructure, asynchronous http, image bundling, monolithic JavaScript compilation, easy RPC).Google app engine provides a separate set of Static file servers dedicated to delivering the static files (HTML, CSS, and JavaScript). These servers are optimized to handle requests for static resources. The static files for a given application can be uploaded along with application code along with configuring how static files are served, including the URLs for static files, content types, instruction for browsers to keep copies of the files in a cache for a given amount of time to reduce traffic and speed up rendering of the page.
3. Reliability
This attribute is primarily related to the datastore capabilities provided by Google App engine. The App engine datastore is designed for applications that need to read quickly, ensuring that data remains in consistent state. Unlike traditional databases, the datastore uses a distributed architecture to manage scaling to very large data sets.
The update of a single entity occurs in a transaction in App engine and each transaction is atomic: the transaction either succeeds completely or fails completely, and cannot succeed or fail in smaller pieces. It leaves the entity in consistent state. The app engine datastore natively supports local transactions. An application can read or update multiple entities in a given transaction, but it must tell App engine which entities will be updated when it creates the entities. The application does this by creating entities in entity groups. If a user tries to update an entity while another user’s update of an entity is in progress, the datastore API returns immediately with a concurrency failure exception. App engine uses optimistic concurrency control. Reading the entity never fails due to concurrency; the application just gets the entity in most recent stable state. Multiple reads can be performed in transaction to make sure that the data read in transaction is current and consistent with self. If application is getting lot of concurrency failures, then it’s important to design entity groups.
So the downtime of application will be due to either failure of the Google servers or when the application is taking lot of time to process the request (more than 30 seconds). The first one goes back to availability of Google infrastructure & no one can question presently their uptime and second point goes back to the performance issue of application (which will be discussed in next section).
4. Performance
The Google App engine's main components - "The sandbox environment, the datastore & the services" defines this quality attribute mainly. When the App engine request handler receives the request and identifies the application from the domain name of the address, it selects a server from many possible servers to process the request & this selection is based on which server is most likely to provide a faster response. App engine mitigates the risk of starting the application for every web request by keeping the application in memory as long as possible & when a server needs to reclaim resources, it purges the least recently used app. So distributed scalable architecture comes at the expense of a little performance degradation.
When the application creates new entities and updates existing ones using the datastore API, the call returns with success or failure after creating/updating entities along with updating every corresponding index. This makes queries very fast at the expense of the entity updates. But again Google app engine runtime uses the memcache service for caching the results of frequently performed queries or calculations. The application checks for a cached value, and if the value is not present in cache, then it performs the query or calculation & stores the value in cache for future use.
Also the runtime environment also limits the amount of clock time, CPU use and memory a single request can take. App engine keeps these limits flexible & applies limits to those applications that use up more resources to protect shared resources from “runaway” applications. But the response time for application can also determine the number of requests the application can handle dynamically. These figures can be checked for quotes & limits restriction defined by Google Platform and can be accessed via URLS mentioned in references section.
URLFetch Services, task Queues & cron jobs are also being defined so that web application can respond to web requests quickly & hence better the performance of the web application.
URL Fetch services are being used by app engine applications to access other web services. The service makes HTTP requests to other servers to retrieve pages or interact with web services and this interaction can be made to fetch the URLs in background so that request handler can continue to process the request. But the fetch URLs process must complete during the request handler lifetime.
Task Queues let request handlers describe the work to be done at a later time, outside the scope of a web browser. Queues ensure that every task gets completed eventually and configured at the rate at which queues are processed to spread the workload throughout the day. A queue performs the task by calling a request handler. It can include a data payload provided by the code that created a task and delivered to the task’s handler as an HTTP request.
App engine has Cron Jobs service for executing tasks at specific times of the day. The scheduled tasks can invoke a request handler at a specific time of the day, week or month based on the schedule provided in configuration.
Scalability attribute is considered as one of the selling points for web applications developed on Google App engine as the applications scale automatically. All three components – the sandbox environment, the (scalable) datastore & the services scales independently from each other.
5. Supportability
Testing –
JUnit test cases can be used for testing the services, datastore & task queues. Google app engine also provides access to different versions of the same application at a given moment of time and can be tested parallel by hitting different urls (along with version-id). This feature can be very useful if newer version of application can be tested completing before launching the same to outside world.
Manageability –
Very minimal as management tools to manage the resources used by application is being done by Google itself and report for the same can be accessed via Administration console.
Configurability -
web.xml is the configuration file defaulted for any web application. In addition to this, Google App engine appengine-web.xml needs to be provided before application is deployed or uploaded onto Google Infrastructure. Appengine-web.xml specifies the app's registered application ID and the version identifier of the latest code, and to identify which files in the app's WAR are static files (like images) and which are resource files used by the application.
Install ability –
As easy as it can get to install an application to Google infrastructure using admin console (via browser) or even using the plug-in for development environment.
Portability & Migration –
This can be one of the attribute not working in favor of the most of the platform providers as they provide vendor lock-in for data. Although the existing applications (using java.io, EJB’s, multi-threaded etc) are not the good candidates for migrating into Google App engine platform (as mentioned these ones are limited by platform capabilities. Even though there is vendor-lock in for the data being stored in datastore and need to retrieve the same back in some format so that the application can be hosted on in-premise infrastructure instead of Google platform.
Some of the open source applications are being written to synchronize the data between Google App engine datastore and relational database. For e.g., AppRocket is an open-source replication engine that synchronizes Google App engine datastore and MySQL database.
App engine includes a tool for uploading and downloading data via the remote API. The tool can create new datastore entities using data from a comma separated values (CSV) data file & can even create CSV files with data from the app’s datastore. These remote API comes into two parts – remote API request handler and the tools & the libraries that call the handler. The remote API handlers are part of Java & Python runtime environment whereas the remote Access tools & libraries are only available for Python. These tools & libraries can be used within Java Application via the Java Remote API request handler, but Google is working on it to provide these libraries and tools part of Java runtime too so that it makes the migration and porting of java application easily into Google Infrastructure.
So selection of Google App engine as platform is mainly driven by no-capital cost, pay-per-use model for resources used beyond free quota, scalability, manageability, server less on-premise infrastructure but limited by its sandbox capabilities and quotas, only web application support and on-going activities to add additional features in java runtime.
References
1. http://en.wikipedia.org/wiki/FURPS - FURPS Model
2. http://code.google.com/appengine/docs/quotas.html - Billable Quotas and Fixed Quotas
3. http://code.google.com/p/approcket/ - AppRocket provides live synchronization between Google App Datastore and MySQL
Google App engine (GAE) is a web application hosting service. GAE can serve traditional website content too (such as documents and images), but the environment is especially designed for real-time dynamic web applications.
This article will use the FURPS model for evaluating the following software quality attributes (functional & non-functional) for applications developed using Java runtime for applications:
1. Functionality - Feature set, Capabilities, Generality, Security
2. Usability - Human factors, Aesthetics, Consistency, Documentation
3. Reliability - Frequency/severity of failure, Recoverability, Predictability, Accuracy, Mean time to failure
4. Performance - Speed, Efficiency, Resource consumption, Throughput, Response time, Scalability
5. Supportability - Testability, Extensibility, Adaptability, Maintainability, Compatibility, Configurability, Serviceability, Install ability, Localizability, Portability
1. Functionality
Feature Set - Irrespective of the platform, feature spot holds the prime spot in any software development project & primary concern for software architect/designer/developer. So any web application irrespective of the platform should be able to meet the requirements.
Capabilities & Generality - Also an architect/designer needs to check whether the system capabilities can be met by Google App engine’s core components – Sandbox environment, datastore & the services(for example, applications relying on multi-threading, using lot of network bandwidth, relational database triggers, accessing the in-network data) before choosing Google as platform. Also the Google App engine costs nothing to get started and hence the capital expenditure [CapEx] at the start of the project is negligible (considering the deployment servers only). So this process can be somehow called as server- less topology at on-premise networks.
Each App Engine resource is measured against one of two kinds of quota: a billable quota or a fixed quota. Every application gets an amount of each billable quota for free. Customer will be charged only for the resources web app actually uses, and only for the amount of resources used above the free quota thresholds. Fixed quotas are resource maximums set by App Engine to ensure the integrity of the system. These resources describe the boundaries of the architecture, and all applications are expected to run within the same limits. They ensure that another app that is consuming too many resources will not affect the performance of your app.
The quotas and limits allowed by Google app engine platform needs to be checked too before making a selection & can be found on App Engine’s Site http://code.google.com/appengine/docs/quotas.html . So using the operational expenditure (OpEx) costs are directly proportional to the amount of resources used in a given calendar day. Also only Web Apps (with limited cron functionality) can be developed on Google App engine platform.
Security –
Sandbox environment increases the security feature of web app as application code can’t access the server on which it is running in the traditional sense. An application can read its own files from the file system, but it can’t write to files & it cannot read files that belong to other applications. An application can see/access the environmental variables set by App engine, but manipulations of these variables do not necessarily persist between requests. An application cannot access the networking facilities of the server hardware. App engine features integration with Google Accounts and hence can use the authentication capabilities of Google with minimal coding/configuration. Also If application is using the authentication via Google Accounts, App Engine includes HTTP headers in the response that give the information about the resources used by a given request.
So sandbox environment can act as "Blessing in disguise" with providing secured wrapper to web application but limiting the capabilities of an application & needs to be carefully checked to see if it really meets the needs of the required web application.
2. Usability
Although this concept is somehow holds the same weight age whether web application is developed using Google App engine Platform or not.
By using Google Web Toolkit (GWT) by far the quickest way to deploy a better UI experience full stack web application using the absolute best possible technologies available (Google’s scalable infrastructure, asynchronous http, image bundling, monolithic JavaScript compilation, easy RPC).Google app engine provides a separate set of Static file servers dedicated to delivering the static files (HTML, CSS, and JavaScript). These servers are optimized to handle requests for static resources. The static files for a given application can be uploaded along with application code along with configuring how static files are served, including the URLs for static files, content types, instruction for browsers to keep copies of the files in a cache for a given amount of time to reduce traffic and speed up rendering of the page.
3. Reliability
This attribute is primarily related to the datastore capabilities provided by Google App engine. The App engine datastore is designed for applications that need to read quickly, ensuring that data remains in consistent state. Unlike traditional databases, the datastore uses a distributed architecture to manage scaling to very large data sets.
The update of a single entity occurs in a transaction in App engine and each transaction is atomic: the transaction either succeeds completely or fails completely, and cannot succeed or fail in smaller pieces. It leaves the entity in consistent state. The app engine datastore natively supports local transactions. An application can read or update multiple entities in a given transaction, but it must tell App engine which entities will be updated when it creates the entities. The application does this by creating entities in entity groups. If a user tries to update an entity while another user’s update of an entity is in progress, the datastore API returns immediately with a concurrency failure exception. App engine uses optimistic concurrency control. Reading the entity never fails due to concurrency; the application just gets the entity in most recent stable state. Multiple reads can be performed in transaction to make sure that the data read in transaction is current and consistent with self. If application is getting lot of concurrency failures, then it’s important to design entity groups.
So the downtime of application will be due to either failure of the Google servers or when the application is taking lot of time to process the request (more than 30 seconds). The first one goes back to availability of Google infrastructure & no one can question presently their uptime and second point goes back to the performance issue of application (which will be discussed in next section).
4. Performance
The Google App engine's main components - "The sandbox environment, the datastore & the services" defines this quality attribute mainly. When the App engine request handler receives the request and identifies the application from the domain name of the address, it selects a server from many possible servers to process the request & this selection is based on which server is most likely to provide a faster response. App engine mitigates the risk of starting the application for every web request by keeping the application in memory as long as possible & when a server needs to reclaim resources, it purges the least recently used app. So distributed scalable architecture comes at the expense of a little performance degradation.
When the application creates new entities and updates existing ones using the datastore API, the call returns with success or failure after creating/updating entities along with updating every corresponding index. This makes queries very fast at the expense of the entity updates. But again Google app engine runtime uses the memcache service for caching the results of frequently performed queries or calculations. The application checks for a cached value, and if the value is not present in cache, then it performs the query or calculation & stores the value in cache for future use.
Also the runtime environment also limits the amount of clock time, CPU use and memory a single request can take. App engine keeps these limits flexible & applies limits to those applications that use up more resources to protect shared resources from “runaway” applications. But the response time for application can also determine the number of requests the application can handle dynamically. These figures can be checked for quotes & limits restriction defined by Google Platform and can be accessed via URLS mentioned in references section.
URLFetch Services, task Queues & cron jobs are also being defined so that web application can respond to web requests quickly & hence better the performance of the web application.
URL Fetch services are being used by app engine applications to access other web services. The service makes HTTP requests to other servers to retrieve pages or interact with web services and this interaction can be made to fetch the URLs in background so that request handler can continue to process the request. But the fetch URLs process must complete during the request handler lifetime.
Task Queues let request handlers describe the work to be done at a later time, outside the scope of a web browser. Queues ensure that every task gets completed eventually and configured at the rate at which queues are processed to spread the workload throughout the day. A queue performs the task by calling a request handler. It can include a data payload provided by the code that created a task and delivered to the task’s handler as an HTTP request.
App engine has Cron Jobs service for executing tasks at specific times of the day. The scheduled tasks can invoke a request handler at a specific time of the day, week or month based on the schedule provided in configuration.
Scalability attribute is considered as one of the selling points for web applications developed on Google App engine as the applications scale automatically. All three components – the sandbox environment, the (scalable) datastore & the services scales independently from each other.
5. Supportability
Testing –
JUnit test cases can be used for testing the services, datastore & task queues. Google app engine also provides access to different versions of the same application at a given moment of time and can be tested parallel by hitting different urls (along with version-id). This feature can be very useful if newer version of application can be tested completing before launching the same to outside world.
Manageability –
Very minimal as management tools to manage the resources used by application is being done by Google itself and report for the same can be accessed via Administration console.
Configurability -
web.xml is the configuration file defaulted for any web application. In addition to this, Google App engine appengine-web.xml needs to be provided before application is deployed or uploaded onto Google Infrastructure. Appengine-web.xml specifies the app's registered application ID and the version identifier of the latest code, and to identify which files in the app's WAR are static files (like images) and which are resource files used by the application.
Install ability –
As easy as it can get to install an application to Google infrastructure using admin console (via browser) or even using the plug-in for development environment.
Portability & Migration –
This can be one of the attribute not working in favor of the most of the platform providers as they provide vendor lock-in for data. Although the existing applications (using java.io, EJB’s, multi-threaded etc) are not the good candidates for migrating into Google App engine platform (as mentioned these ones are limited by platform capabilities. Even though there is vendor-lock in for the data being stored in datastore and need to retrieve the same back in some format so that the application can be hosted on in-premise infrastructure instead of Google platform.
Some of the open source applications are being written to synchronize the data between Google App engine datastore and relational database. For e.g., AppRocket is an open-source replication engine that synchronizes Google App engine datastore and MySQL database.
App engine includes a tool for uploading and downloading data via the remote API. The tool can create new datastore entities using data from a comma separated values (CSV) data file & can even create CSV files with data from the app’s datastore. These remote API comes into two parts – remote API request handler and the tools & the libraries that call the handler. The remote API handlers are part of Java & Python runtime environment whereas the remote Access tools & libraries are only available for Python. These tools & libraries can be used within Java Application via the Java Remote API request handler, but Google is working on it to provide these libraries and tools part of Java runtime too so that it makes the migration and porting of java application easily into Google Infrastructure.
So selection of Google App engine as platform is mainly driven by no-capital cost, pay-per-use model for resources used beyond free quota, scalability, manageability, server less on-premise infrastructure but limited by its sandbox capabilities and quotas, only web application support and on-going activities to add additional features in java runtime.
References
1. http://en.wikipedia.org/wiki/FURPS - FURPS Model
2. http://code.google.com/appengine/docs/quotas.html - Billable Quotas and Fixed Quotas
3. http://code.google.com/p/approcket/ - AppRocket provides live synchronization between Google App Datastore and MySQL
Subscribe to:
Posts (Atom)