How to deal with overposting/underposting when your setup is OData with Entity Framework

Question

The normal path in WEB API is to have your "Entity" and the "DTO".

For example:

public class Product  // DB Entity
{
    public int Id { get; set; }
    public string Name { get; set; }
    public string SerialNumber { get; set; }
    public string ReferenceNumber { get; set; }
    public DateTime RecordTimestamp { get; set; }
    public string InternalNumber { get; set; }
}

public class ProductDto  // Client-facing DTO
{
    public string Name { get; set; }
    public string SerialNumber { get; set; }
    public string ReferenceNumber { get; set; }
} 

In this case, not just posting but also viewing is the issue. We want consumer to see part of the data and hide system data. But at the same time, this same Context is used for POSTing (by permitted accounts), hence entity models must have the properties that can be used for CRUD.

The normal OData API is set as following - Controller is deriving from ODataController and GET actions return IQueryable<T> where this conquerable is routed to DbSet<T>

OData controller does all the magic when OData query is passed. But how in this case we could hide the properties we don't want to display in GET, if the DTO model is same as Entity model.

Is there a way to do this via attributes (model binding?) or this case requires different entities for POST vs GET? But even if I do create separate entity for GET I still need to have Key fields because ODAta uses these attributes to build its relations.

Answer 1

Regarding DTOs, the OData Model IS a DTO mapping for ALL of the types that you expose. Unless your DTOs significantly change the structure of the data then it is not necessary to use other ORM tools like AutoMapper or to manually map the OData queries and Model artifacts to their EF counterparts, or to expose your own DTOs for business entities. You are expected to do this in the OData Model definition itself.

DTOs are useful for complex queries where you want to present an entirely different schema to the end client that might be better suited to a specific process or user experience, but not required for CRUD data entry management.

How to prevent Over-Posting in OData

Overposting meaning that more properties were set than was intended, potentially allowing internal use or status fields like a Balance or Limit fields or Appproval flags being manipulated deliberately by the client.

If certain fields are not required by the client at all, then we can remove these fields from the OData Schema altogether, effectively creating a DTO definition that omits these properties.

The first principals approach is to deliberately exclude the specific fields using the OData Fluent Configuration API after you have registered the types and bound them to EntitySets:

var productConfig = builder.EntityType<Product>();
productConfig.Ignore(x => x.RecordTimestamp);
productConfig.Ignore(x => x.InternalNumber);

Unfortunately, in .Net FX these Ignore methods are not chainable... We also cannot use NotMappedAttribute as this would specifically remove this field from the EF context and therefor the database.

Do not exclude the Key properties from the OData Model, to do so would significantly complicate the addressing of individual resources and in many cases break the OData runtime.

As detailed in this post you can use IgnoreDataMemberAttribute on the properties to explicitly exclude them from the OData Model if you are using the ODataConventionModelBuilder. This is how that would look:

public class Product  // DB Entity
{
    public int Id { get; set; }
    public string Name { get; set; }
    public string SerialNumber { get; set; }
    public string ReferenceNumber { get; set; }
    [IgnoreDataMember]
    public DateTime RecordTimestamp { get; set; }
    [IgnoreDataMember]
    public string InternalNumber { get; set; }
}

How to prevent Under-Posting

As with MVC implementations, the RequiredAttribute can be used to require that specific fields are included in a POST/PATCH against a specific entity. But it will only be automatically applied if the ODataConventionModelBuilder is used, otherwise you will need to set this explicitly:

var productConfig = builder.EntityType<Product>();
productConfig.HasRequired(x => x.Name);

If you are using the ODataConventionModelBuilder you can annotate the model directly:

public class Product  // DB Entity
{
    public int Id { get; set; }
    [Required]
    public string Name { get; set; }
    public string SerialNumber { get; set; }
    public string ReferenceNumber { get; set; }
    [IgnoreDataMember]
    public DateTime RecordTimestamp { get; set; }
    [IgnoreDataMember]
    public string InternalNumber { get; set; }
}

When factoring or testing for Under Posting remember that OData PATCH is specifically designed for the client to send only the properties that should be changed, properties that are not provided in the request for a PATCH should remain unaffected, but the request should still be processed as a valid request as long as all the required properties are included.

Conditional Logic

You can also explicitly validate Under or Over post constraints in the PATCH handler in your controller implementation. This can be useful for implementing discretionary based validation logic, perhaps dependent on security rules or roles assigned to the user. It is also useful if you want the internal values to be utilised in the client for some operations, perhaps to be readonly, or you want to only allow certain fields to be updated through a specific endpoint that has other criteria or security constraints of its own.

/// <summary>
/// Update an existing item with a deltafied or partial declared JSON object
/// </summary>
/// <param name="key">The ID of the item that we want to update</param>
/// <param name="patch">The deltafied or partial representation of the fields that we want to update</param>
/// <param name="options">Parsed OData query options, not used directly but assists method pattern matching for swagger doc generation</param>
/// <remarks>PATCH: odata/DataItems(5) { "Property" : "Value" }</remarks>
/// <returns>UpdatedOdataResult</returns>
[EnableQuery(AllowedQueryOptions = AllowedQueryOptions.DeltaToken | AllowedQueryOptions.Format | AllowedQueryOptions.Select)]
[AcceptVerbs("PATCH", "MERGE")]
public virtual async Task<IHttpActionResult> Patch([FromODataUri] TKey key, Delta<TEntity> patch, ODataQueryOptions<TEntity> options)
{
    // Validate RequiredAttribute and other Edm Model constraints
    if (!ModelState.IsValid)
        return BadRequest(ModelState);

    if (patch == null)
        throw new ArgumentNullException("patch", "Ensure that incoming structure is valid JSON and that you do not attempt to patch nested properties, this controller does not support that");

    var itemQuery = db.Products.Where(x => x.Id == key);

    // simple concurrency check - only works if ETag is provided in the request
    if (options.IfMatch != null &&
        !options.IfMatch.ApplyTo(itemQuery).Any())
    {
        return StatusCode(HttpStatusCode.PreconditionFailed);
    }

    var item = itemQuery.FirstOrDefault();
    if (item == null)
    {
        return NotFound();
    }

    // TODO: validate conditional Under/Over POST constraints
    //       or constraints not declared in the Edm Model

    // don't allow edits to Name via this interface
    if (patch.GetChangedPropertyNames().Contains())
        return BadRequest($"{nameof(Product.Name)} cannot be modified via PATCH, please use the RenameProduct action instead");

    // Apply the properties to the underlying object
    patch.Patch(item);

    // Validate that one of SerialNumber OR ReferenceNumber are specified.
    // This does not mean that they were provided in the PATCH, only that after the PATCH, in the data model at least one of these properties has a value
    if (String.IsNullOrEmpty(item.SerialNumber) && String.IsNullOrEmpty(item.ReferenceNumber))
        return BadRequest($"One of {nameof(Product.SerialNumber)} OR {nameof(Product.ReferenceNumber)} must have a value!");

    // After validation, commit the change to the database
    try
    {
        await db.SaveChangesAsync();
    }
    catch (DbUpdateConcurrencyException)
    {
        if (itemQuery.Count() == 0)
            return NotFound();
        else
            throw;
    }

    return Updated(item);
}

This example is more verbose than many, but it is important to showcase the standard implementations, rather than only part of it, as is common in documentation. I hope this drives follow up questions ;)

---

More on DTOs

We sometimes see examples on-line of OData implementations that use tools like AutoMapper instead of properly configuring the OData Model. Sometimes this is simply due to the configuration logic being sometimes cumbersome in comparison to AutoMapper. Other instances are examples of API (or developers) that have transitioned from a previous business domain repository that already implemented AutoMapper. Sometimes it is simply that the functionality is not well understood.

The ODataConventionModelBuilder was not designed to allow developers the opportunity to pick and choose the specific conventions that are implemented, nor does it allow you to add your own custom conventions OOTB. That has made it harder for the community to adapt to configuration in this space and due to under-representation in production workloads and therefore support queries, there is minimal reference to this in the documentation provided.

If you do manually map your DTOs to the EF context, then the OData Model DTOs will represent a 1:1 mapping of your internal DTOs to the external callers. There is a negligible performance impact to this, but it is an additional layer of configuration and therefore management that in many cases is not required for an OData API.

---

If Over/Under Post is not a concern

If the main driver for a DTO is performance then OData solution is to implement a $select query parameter to only return the fields that you need, we do not need to implement a specific DTO for this.

Optimizing Web Applications with OData $Select
OData $select enables API consumers to shape the data they are consuming before the data is returned from the API endpoint they are calling.

This Allows the client to constrain the data that should be returned, but allows the server to implement the logic to do this, primarily to reduce the bytes transmitted between the server and the client.

An EF backed controller implementation that utilizes IQueryable allows this request for specific fields to be deferred to the backing data store in the form of a modified SQL query, only transmitting these fields from the database to the API and then to the client. Giving us improved response from the database and the API.

When the calling client knows the specific fields that it needs to display, then it is appropriate to allow the client to manage the $select, in this case $select=Name,SerialNumber,ReferenceNumber

You can also manage the default value of $select to apply to all queries for each type where the client does NOT specify a $select. However, if you do this, your client will need to explicitly call $select=* to ensure that all fields are returned, or specifically include the key fields, in this case Id if you do not already have that information.

NOTE: If your create or update logic from the client does not directly provide values for the hidden fields and does not need them to drive UI logic, then we can remove them altogether from the OData context. The server implementation still has access to the whole EF schema to process CRUD requests through the OData API, there is no need to include them in the client context if you do not need them in the client!