duplicate key error index when doing an upsert

I have some (C#) code that upserts a bunch of documents as follows:

        public void UpsertByProductID(IEnumerable<Product> documents)

        {

            if (documents.Any())

            {

                var query = Query.In("Attributes.Product ID", documents.Select(x => BsonValue.Create(x["Product ID"])));

                productsCollection.Remove(query, RemoveFlags.None, WriteConcern.Acknowledged); 

                productsCollection.InsertBatch(documents);

            }

        }

And I'm regularly getting the following error:

WriteConcern detected an error ''. (Response was { "err" : "E11000 duplicate key error index: skubrain.Product.$Attributes.Product ID_1  dup key: { : null }", "code" : 11000, "n" : 0, "connectionId" : 11, "ok" : 1.0 }).

There is (obviously) a unique index defined on the  Attributes.Product ID field. However as far as I can tell, from the above logic, it shouldn't be possible for this code to cause this write concern error... it is specifically deleting any documents that match the Attributes.Product ID of the documents being inserted before performing the insert. 

Any ideas?

I'm regularly getting the following error:

WriteConcern detected an error ''. (Response was { "err" : "E11000 duplicate key error index: mydatabase.Product.$Attributes.Product ID_1  dup key: { : null }", "code" : 11000, "n" : 0, "connectionId" : 11, "ok" : 1.0 }).

There is (obviously) a unique index defined on the  Attributes.Product ID field. However as far as I can tell, from the above logic, it shouldn't be possible for this code to cause this write concern error... it is specifically deleting any documents that match the Attributes.Product ID of the documents being inserted before performing the insert. 

The duplicate key error indicates that the issue is with "null" values for "Attributes.Product ID".

If a document doesn't have a value for the field in a unique index, the index entry will be null unless the index is both unique and sparse:

 http://docs.mongodb.org/manual/tutorial/create-a-unique-index/

In a non-sparse unique index, that means that only one document in the collection can have a null/missing value.

I would check that your documents either always include the "Attributes.Product ID" field, or drop and recreate the unique index as unique & sparse if the "Product ID" is not always present.

OK so I've got no idea why the bulk insert didn't work but I've basically worked around it for the time with a loop that upserts documents individually:

       public void UpsertByProductID(IEnumerable<Product> documents)

        {

            if (documents.Any())

            {

                foreach (var doc in documents)

                    try

                    {

                        var query = Query.EQ("Attributes.Product ID", BsonValue.Create(doc["Product ID"]));

                        var existing = FindOne(query);

                        if (existing != null)

                            doc.Id = existing.Id;

                        Update(

                            query, 

                            Update<Product>.Replace(doc), 

                            UpdateFlags.Upsert

                            );

                    }

                    catch (Exception ex)

                    {

                        ex.Data["Json"] = doc.ToJson();

                        throw ex;

                    }

            }

        }

It's a bit more wordy that my original code but it has the incomparable advantage that it works...

I noticed that as well... however the error message was erroneous in that respect since:

a) the index was defined as sparse and

b) none of the documents being inserted had a null or otherwise blank (e.g. whitespace) value for Attributes.Product ID. 

I initially tried replacing the bulk insert with a loop that used FindAndModify to individually upsert each of the documents and I was still getting the error... so I knew exactly which document had caused the error. Something was really odd about it. If I took the JSON for that document and simply did a find in MongoVue for any documents matching the index fields, I was getting nothing back. Also, if I performed the Upsert from within MongoVue everything went swimmingly.

I suspect then that it might be an issue with the C# driver. In any event, using Update (with upsert = true) instead of FindAndModify appears to have resolved the issue for the time being.