berkeley LockTimeoutException.

Hi Guys,
I'm continuing with my testing. The following works with cassandra-es but if I use titan-berkeley-es.properties I get the following error.
I'm trying to understand which timeout this is and if it's something I need to change in the configuration or not. But I can't seem to figure it out:
The error is as follows. The app does a few things:

opens a session
executes some code (can take some time)
submits queries for the session.

We can see roughly that 1500ms pass between the session creation and script submission then the lock timeout happens. Any help would be appreciated.
(this works with cassandra-es)
1220682 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.server.op.session.Session  - New session established for b150ab0a-1424-4b42-9361-d8e6bd4aabe6
1220708 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines  - Loaded nashorn ScriptEngine
1220721 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines  - Loaded gremlin-groovy ScriptEngine
1221081 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor  - Initialized gremlin-groovy ScriptEngine with scripts/empty-sample.groovy
1222523 [pool-98-thread-1] ERROR com.thinkaurelius.titan.graphdb.database.StandardTitanGraph  - Could not commit transaction [94] due to storage exception in commit
com.thinkaurelius.titan.core.TitanException: Could not execute operation due to backend exception
    at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:44)
    at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:87)
    at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:141)
    at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:198)
    at com.thinkaurelius.titan.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:118)
    at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:717)
    at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1329)
    at com.thinkaurelius.titan.graphdb.tinkerpop.TitanBlueprintsGraph$GraphTransaction.doCommit(TitanBlueprintsGraph.java:263)
    at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:94)
    at org.apache.tinkerpop.gremlin.structure.Transaction$commit$2.call(Unknown Source)
    at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110)
    at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:114)
    at Script4.run(Script4.groovy:1)
    at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:447)
    at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:293)
    at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
    at org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines.eval(ScriptEngines.java:89)
    at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$2(GremlinExecutor.java:244)
    at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor$$Lambda$108/1268451829.call(Unknown Source)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.thinkaurelius.titan.diskstorage.PermanentBackendException: Permanent failure in storage backend
    at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEKeyValueStore.insert(BerkeleyJEKeyValueStore.java:206)
    at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEKeyValueStore.insert(BerkeleyJEKeyValueStore.java:183)
    at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEStoreManager.mutateMany(BerkeleyJEStoreManager.java:195)
    at com.thinkaurelius.titan.diskstorage.keycolumnvalue.keyvalue.OrderedKeyValueStoreManagerAdapter.mutateMany(OrderedKeyValueStoreManagerAdapter.java:107)
    at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:90)
    at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:87)
    at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56)
    at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42)
    ... 23 more
Caused by: com.sleepycat.je.LockTimeoutException: (JE 5.0.73) Lock expired. Locker<PHONE_NUMBER> 1873_pool-98-thread-1_Txn: waited for lock on database=edgestore LockAddr:313929167 LSN=0x0/0x2708f type=WRITE grant=WAIT_PROMOTION timeoutMillis=500 startTime=1435142947866 endTime=1435142948366
Owners: [<LockInfo locker="104208586 1872_gremlin-server-worker-1_Txn" type="READ"/>, <LockInfo locker="2094710972 1873_pool-98-thread-1_Txn" type="READ"/>]
Waiters: []
Transaction<PHONE_NUMBER> 1873_pool-98-thread-1_Txn owns LockAddr:313929167 <LockInfo locker="2094710972 1873_pool-98-thread-1_Txn" type="READ"/>
Transaction<PHONE_NUMBER> 1873_pool-98-thread-1_Txn waits for LockAddr:313929167

    at com.sleepycat.je.txn.LockManager.newLockTimeoutException(LockManager.java:664)
    at com.sleepycat.je.txn.LockManager.makeTimeoutMsgInternal(LockManager.java:623)
    at com.sleepycat.je.txn.SyncedLockManager.makeTimeoutMsg(SyncedLockManager.java:97)
    at com.sleepycat.je.txn.LockManager.lockInternal(LockManager.java:390)
    at com.sleepycat.je.txn.LockManager.lock(LockManager.java:276)
    at com.sleepycat.je.txn.Txn.lockInternal(Txn.java:498)
    at com.sleepycat.je.txn.Locker.lock(Locker.java:443)
    at com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:2621)
    at com.sleepycat.je.dbi.CursorImpl.getSlotReuseInfo(CursorImpl.java:1091)
    at com.sleepycat.je.dbi.CursorImpl.putInternal(CursorImpl.java:880)
    at com.sleepycat.je.dbi.CursorImpl.put(CursorImpl.java:811)
    at com.sleepycat.je.Cursor.putAllowPhantoms(Cursor.java:2243)
    at com.sleepycat.je.Cursor.putNoNotify(Cursor.java:2200)
    at com.sleepycat.je.Cursor.putNotify(Cursor.java:2117)
    at com.sleepycat.je.Cursor.putNoDups(Cursor.java:2052)
    at com.sleepycat.je.Cursor.putInternal(Cursor.java:2020)
    at com.sleepycat.je.Database.putInternal(Database.java:1302)
    at com.sleepycat.je.Database.put(Database.java:1172)
    at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEKeyValueStore.insert(BerkeleyJEKeyValueStore.java:194)
    ... 30 more

Hmm - not sure about this one.  you say "executes some code" - do you mean in scripts/empty-sample.groovy?  if so, is it possible that it's still running when you make your request somehow (i thought that Gremlin Server would prevent that)?

No, just some application code.
I start the transaction and session.
Then I extract meta data from some files and feed them to gremlin-server via gremlin queries (adding a vertex and/or changing/adding properties to other vertices).
I commit the transaction.
I initially thought the settings for berkeley made the transaction timeout during the time it took to extract the meta data. But I'm not sure that's a good interpretation.
As extra info there shouldn't be any concurrency on this operation (at least not within the testing scenario).
I'm still trying to narrow down on my random failures, figured I would try using berkeley in place of cassandra to see if they still occurred.

I initially thought the settings for berkeley made the transaction timeout during the time it took to extract the meta data. But I'm not sure that's a good interpretation.

I don't think so - if you haven't issue a request then simply opening the session shouldn't cause a transaction timeout (not so sure that a transaction is officially started at that point without a request).  You probably need to provide the nature of the requests you are sending for me to say more.

I'll have a look at the code to make sure but I think it's safe to assume that the session is opened with a query of graph.tx().open(). I just thought it was weird that it failed for berkeley but worked fine with cassandra.

so you are sending manual transaction management commands?  by default, you don't need to issue an open() unless you've configured it for MANUAL transactions.  of course, you may be on to bug in MANUAL so this probably isn't a bad thing to explore a bit.

Yeah let me go a bit more into detail.
The way the driver works is that if you start a transaction is will send graph.tx().open() with an op of session to gremlin-server. Thus configuring for MANUAL transactions and opening one at the same time.
That's the situation here in this use case.

We create a session so entering MANUAL handling of transactions
We start a transaction with graph.tx().open() (both these steps are one and the same)
We spend time extracting meta data
We mutate the graph with the data we got
We commit with graph.tx().commit()
Error ensues.

@PommeVerte I definitely lost track of this one until @mbroecheler assigned it to me a few hours ago.  is this still an issue?

I'll give it another go. Should I build against titan09?

yes - titan09 please.

Sorry I've been off work. Finally found time for this and should be more present overall. Quick question as I seem to be running into some issues running the same series of tests. Is key a reserved property name now?

Yes, "key" is now a reserved word in the type system on the titan09 / 1.0 branch (https://github.com/thinkaurelius/titan/issues/634)

Ok that's a bummer. It means I'll have to refactor both the core and frontend of the software before I can provide more input on this error so it might be a little while.
Was that the only one added? Or are there any other plans for reserved words? I'll probably be changing key for uid given the circumstances.

Ok so I took the time to refactor and look into this again and I'm still getting this issue. I placed the error at the end of this post.
I might have more time today to try and recreate this but in essence here's what I think the issue is (all is done via gremlin-server with session OpProcessor):

Open a transaction (graph.tx().start())
Wait x seconds (from the error message I'm guessing 500ms)
Mutate graph
Commit
get timeout error.

(2. and 3. may be permutable, I'm not sure if the mutations reset the timeout counter)
This might be something I could fix via configuration, the thing is I don't have this problem when using cassandra, so there's definitely some inconsistency in how this is set up by default.
934863 [pool-113-thread-1] WARN  org.apache.tinkerpop.gremlin.server.op.AbstractEvalOpProcessor  - Exception processing a script on request [RequestMessage{, requestId=89355315-74b3-49f0-852c-d059a5887a56, op='eval', processor='session', args={gremlin=graphT.tx().commit(), session=2fc4b136-3611-4719-8890-1a8f4da22156}}].
com.sleepycat.je.LockTimeoutException: (JE 5.0.73) Lock expired. Locker<PHONE_NUMBER> 2699_pool-113-thread-1_Txn: waited for lock on database=edgestore LockAddr:305482869 LSN=0x0/0x18e855 type=WRITE grant=WAIT_PROMOTION timeoutMillis=500 startTime=1442338472093 endTime=1442338472593
Owners: [<LockInfo locker="322570170 2698_gremlin-server-worker-1_Txn" type="READ"/>, <LockInfo locker="2073414680 2699_pool-113-thread-1_Txn" type="READ"/>]
Waiters: []
Transaction<PHONE_NUMBER> 2699_pool-113-thread-1_Txn owns LockAddr:305482869 <LockInfo locker="2073414680 2699_pool-113-thread-1_Txn" type="READ"/>
Transaction<PHONE_NUMBER> 2699_pool-113-thread-1_Txn waits for LockAddr:305482869

	at com.sleepycat.je.txn.LockManager.newLockTimeoutException(LockManager.java:664)
	at com.sleepycat.je.txn.LockManager.makeTimeoutMsgInternal(LockManager.java:623)
	at com.sleepycat.je.txn.SyncedLockManager.makeTimeoutMsg(SyncedLockManager.java:97)
	at com.sleepycat.je.txn.LockManager.lockInternal(LockManager.java:390)
	at com.sleepycat.je.txn.LockManager.lock(LockManager.java:276)
	at com.sleepycat.je.txn.Txn.lockInternal(Txn.java:498)
	at com.sleepycat.je.txn.Locker.lock(Locker.java:443)
	at com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:2621)
	at com.sleepycat.je.dbi.CursorImpl.getSlotReuseInfo(CursorImpl.java:1091)
	at com.sleepycat.je.dbi.CursorImpl.putInternal(CursorImpl.java:880)
	at com.sleepycat.je.dbi.CursorImpl.put(CursorImpl.java:811)
	at com.sleepycat.je.Cursor.putAllowPhantoms(Cursor.java:2243)
	at com.sleepycat.je.Cursor.putNoNotify(Cursor.java:2200)
	at com.sleepycat.je.Cursor.putNotify(Cursor.java:2117)
	at com.sleepycat.je.Cursor.putNoDups(Cursor.java:2052)
	at com.sleepycat.je.Cursor.putInternal(Cursor.java:2020)
	at com.sleepycat.je.Database.putInternal(Database.java:1302)
	at com.sleepycat.je.Database.put(Database.java:1172)
	at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEKeyValueStore.insert(BerkeleyJEKeyValueStore.java:194)
	at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEKeyValueStore.insert(BerkeleyJEKeyValueStore.java:183)
	at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEStoreManager.mutateMany(BerkeleyJEStoreManager.java:196)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.keyvalue.OrderedKeyValueStoreManagerAdapter.mutateMany(OrderedKeyValueStoreManagerAdapter.java:107)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:90)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:87)
	at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56)
	at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:87)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:141)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:198)
	at com.thinkaurelius.titan.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:118)
	at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:717)
	at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1352)
	at com.thinkaurelius.titan.graphdb.tinkerpop.TitanBlueprintsGraph$GraphTransaction.doCommit(TitanBlueprintsGraph.java:263)
	at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:94)
	at org.apache.tinkerpop.gremlin.structure.Transaction$commit$2.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:114)
	at Script4.run(Script4.groovy:1)
	at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:525)
	at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:365)
	at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
	at org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines.eval(ScriptEngines.java:96)
	at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$22(GremlinExecutor.java:244)
	at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor$$Lambda$121/412658246.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

Ok So I think I'm getting the gist of this, it might be an issue with gremlin server (not entirely sure though). Cassandra isn't ACID so it will not complain about this. However berkeley is and there must be some sort of concurrent conflict over the lock. With that said here's a more in depth explanation of what is happening:

Our software opens a session (needs to manually set a transaction).
opens transaction
mutates the graph (vertex X)
commit transaction
another thread is spawned and does the following (we execute this synchronously for testing purposes):

opens a new session (can't help it)
opens transaction
mutates the graph (vertex X)
commit transaction <-- error is produced because we're mutating the same element
closes session

closes session

The error shouldn't be thrown as there are no concurrent transactions running. I've even set delays to make sure transactions have the time to close properly.
Which I think could mean one of three things:

I missed a place where there could be an overlap (normally I haven't but I will double check)
Committing a transaction doesn't free the lock (are locks freed on transaction.close()?)
Sessions somehow interfere via some rogue transaction hanging around (which I don't think is the case seeing as transactions need to be manually handled with the session opProcessor).

I'm placing my money on 2)

You should be calling commit() or rollback() - the behavior of close() is dependent on the onClose() behavior which we've found is a bit shaky in its current implementation on the TP3 side.  I would expect locks to be freed on commit/rollback() and that you should not call close() at all for what you are describing.
As you specifically stated that this is a synchronous process, we should be able to recreate it in the console.  Unfortunately, I can't seem to do that - i think this is a pretty good approximation for what you described:
gremlin> graph = TitanFactory.open('conf/titan-berkeleyje.properties')
==>standardtitangraph[berkeleyje:/home/smallette/jvm/titan-1.0.0-SNAPSHOT-hadoop1/conf/../db/berkeley]
gremlin> graph.tx().onReadWrite(Transaction.READ_WRITE_BEHAVIOR.MANUAL)
==>com.thinkaurelius.titan.graphdb.tinkerpop.TitanBlueprintsGraph$GraphTransaction@760245e1
gremlin> mgmt = graph.openManagement()
==>com.thinkaurelius.titan.graphdb.database.management.ManagementSystem@534c6767
gremlin> name = mgmt.makePropertyKey("name").dataType(String.class).make()
==>name
gremlin> namei =  mgmt.buildIndex("name", Vertex.class).addKey(name).unique().buildCompositeIndex()
==>name
gremlin> mgmt.setConsistency(namei, ConsistencyModifier.LOCK)
==>null
gremlin> mgmt.commit()
==>null
gremlin> graph.tx().open()
==>null
gremlin> graph.addVertex('name','rick')
==>v[4288]
gremlin> graph.tx().commit()
==>null
gremlin> graph.tx().open()
==>null
gremlin> g = graph.traversal()
==>graphtraversalsource[standardtitangraph[berkeleyje:/db/berkeley], standard]
gremlin> Thread.start{
gremlin>   g.tx().open()
gremlin>   def v = g.V().has('name','rick').next()
gremlin>   v.property('name','bob')
gremlin>   g.tx().commit()
gremlin> }
==>Thread[Thread-9,5,main]
gremlin> g.tx().open()
==>null
gremlin> g.V().has('name','rick')
gremlin> g.V().has('name','bob')
==>v[4288]

Guess I need to try with Gremlin Server next.............

I can't recreate this in Gremlin Server either:
gremlin> sessionA.submit("graph.tx().onReadWrite(Transaction.READ_WRITE_BEHAVIOR.MANUAL);graph.tx().rollback()")
gremlin> sessionB.submit("graph.tx().onReadWrite(Transaction.READ_WRITE_BEHAVIOR.MANUAL);graph.tx().rollback()")
gremlin> sessionA.submit("graph.tx().open()").all().get()
==>result{object=null class=null}
gremlin> sessionA.submit("graph.vertices(4288).next().property('name','alice');null").all().get()
==>result{object=null class=null}
gremlin> sessionA.submit("graph.tx().commit()").all().get()
==>result{object=null class=null}
gremlin> sessionB.submit("graph.tx().open()").all().get()
==>result{object=null class=null}
gremlin> sessionB.submit("graph.vertices(4288).next().property('name','al');null").all().get()
==>result{object=null class=null}
gremlin> sessionB.submit("graph.tx().commit()").all().get()
==>result{object=null class=null}

I didn't do the "new thread" thing on the client side as that should't really matter to what happens on the server.
I can recreate this error if I start mixing stuff up:
gremlin> sessionA.submit("graph.tx().open()").all().get()
==>result{object=null class=null}
gremlin> sessionB.submit("graph.tx().open()").all().get()
==>result{object=null class=null}
gremlin> sessionB.submit("graph.vertices(4288).next().property('name','al');null").all().get()
==>result{object=null class=null}
gremlin> sessionA.submit("graph.vertices(4288).next().property('name','alice');null").all().get()
==>result{object=null class=null}
gremlin> sessionA.submit("graph.tx().commit()").all().get()
java.lang.RuntimeException: org.apache.tinkerpop.gremlin.driver.exception.ResponseException: (JE 5.0.73) Lock expired. Locker<PHONE_NUMBER> 1087_pool-9-thread-1_Txn: waited for lock on database=graphindex LockAddr:1550945149 LSN=0x0/0x47b4 type=WRITE grant=WAIT_PROMOTION timeoutMillis=500 startTime=1442497793161 endTime=1442497793663
Owners: [<LockInfo locker="267138345 1088_pool-14-thread-1_Txn" type="READ"/>, <LockInfo locker="1270788668 1087_pool-9-thread-1_Txn" type="READ"/>]
Waiters: []
Transaction<PHONE_NUMBER> 1087_pool-9-thread-1_Txn owns LockAddr:1550945149 <LockInfo locker="1270788668 1087_pool-9-thread-1_Txn" type="READ"/>
Transaction<PHONE_NUMBER> 1087_pool-9-thread-1_Txn waits for LockAddr:1550945149

But that's pretty much to be expected because two different transactions are trying to modify the same property on the same vertex.  The user would need to rollback and retry the transactions.
My best guess is that your synchronous behavior is not as synchronous as you might think?  Do you have some way of recreating this with the console in the manner that I tried to do here?

Thanks for taking the time to test all that out Stephen. This is turning out to be quite a headache. I think there might be two separate issues here.
I've noticed that if I get the error and then just kill the app without closing the sockets/session then the lock persists (indefinitely) and I need to reboot the server and reset the database entirely (rm -rf db/berkeley).
If I only reboot and don't reset the database I get the following error and warning for mutating the locked element(s) (in this case deleting all vertices from within a transaction):
15294 [pool-9-thread-1] ERROR com.thinkaurelius.titan.graphdb.database.StandardTitanGraph  - Could not commit transaction [1] due to storage exception in commit
com.thinkaurelius.titan.core.TitanException: Could not execute operation due to backend exception
	at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:44)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:87)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:141)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:198)
	at com.thinkaurelius.titan.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:118)
	at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:717)
	at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1352)
	at com.thinkaurelius.titan.graphdb.tinkerpop.TitanBlueprintsGraph$GraphTransaction.doCommit(TitanBlueprintsGraph.java:263)
	at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:94)
	at org.apache.tinkerpop.gremlin.structure.Transaction$commit$1.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:114)
	at Script4.run(Script4.groovy:1)
	at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:525)
	at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:365)
	at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
	at org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines.eval(ScriptEngines.java:96)
	at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$22(GremlinExecutor.java:244)
	at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor$$Lambda$126/217699241.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: com.thinkaurelius.titan.diskstorage.PermanentBackendException: Could not remove: OperationStatus.NOTFOUND
	at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEKeyValueStore.delete(BerkeleyJEKeyValueStore.java:219)
	at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEStoreManager.mutateMany(BerkeleyJEStoreManager.java:202)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.keyvalue.OrderedKeyValueStoreManagerAdapter.mutateMany(OrderedKeyValueStoreManagerAdapter.java:107)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:90)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:87)
	at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56)
	at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42)
	... 23 more
15298 [pool-9-thread-1] ERROR com.thinkaurelius.titan.graphdb.database.StandardTitanGraph  - Could not commit transaction [1] due to exception
com.thinkaurelius.titan.core.TitanException: Could not execute operation due to backend exception
	at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:44)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:87)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:141)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:198)
	at com.thinkaurelius.titan.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:118)
	at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:717)
	at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1352)
	at com.thinkaurelius.titan.graphdb.tinkerpop.TitanBlueprintsGraph$GraphTransaction.doCommit(TitanBlueprintsGraph.java:263)
	at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:94)
	at org.apache.tinkerpop.gremlin.structure.Transaction$commit$1.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:114)
	at Script4.run(Script4.groovy:1)
	at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:525)
	at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:365)
	at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
	at org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines.eval(ScriptEngines.java:96)
	at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$22(GremlinExecutor.java:244)
	at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor$$Lambda$126/217699241.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: com.thinkaurelius.titan.diskstorage.PermanentBackendException: Could not remove: OperationStatus.NOTFOUND
	at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEKeyValueStore.delete(BerkeleyJEKeyValueStore.java:219)
	at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEStoreManager.mutateMany(BerkeleyJEStoreManager.java:202)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.keyvalue.OrderedKeyValueStoreManagerAdapter.mutateMany(OrderedKeyValueStoreManagerAdapter.java:107)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:90)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:87)
	at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56)
	at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42)
	... 23 more
15309 [pool-9-thread-1] WARN  org.apache.tinkerpop.gremlin.server.op.AbstractEvalOpProcessor  - Exception processing a script on request [RequestMessage{, requestId=ce0ed4e0-2a2d-4a93-8cfa-049388bc1ee8, op='eval', processor='session', args={gremlin=graphT.tx().commit(), session=131fa607-0a06-4921-a8f4-4278a80ab0e6}}].
com.thinkaurelius.titan.diskstorage.PermanentBackendException: Could not remove: OperationStatus.NOTFOUND
	at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEKeyValueStore.delete(BerkeleyJEKeyValueStore.java:219)
	at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEStoreManager.mutateMany(BerkeleyJEStoreManager.java:202)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.keyvalue.OrderedKeyValueStoreManagerAdapter.mutateMany(OrderedKeyValueStoreManagerAdapter.java:107)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:90)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:87)
	at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56)
	at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:87)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:141)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:198)
	at com.thinkaurelius.titan.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:118)
	at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:717)
	at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1352)
	at com.thinkaurelius.titan.graphdb.tinkerpop.TitanBlueprintsGraph$GraphTransaction.doCommit(TitanBlueprintsGraph.java:263)
	at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:94)
	at org.apache.tinkerpop.gremlin.structure.Transaction$commit$1.call(Unknown Source)
	at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:110)
	at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:114)
	at Script4.run(Script4.groovy:1)
	at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:525)
	at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:365)
	at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
	at org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines.eval(ScriptEngines.java:96)
	at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$22(GremlinExecutor.java:244)
	at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor$$Lambda$126/217699241.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

I would expect the offending transaction's session to eventually timeout, roll the transaction back and free the lock. Let alone have it persist after a reboot.
I can't seem to figure out what is persisting here, if it's the gremlin-server session (with an open transaction) or if it's some titan locking mechanism that just never lets go.

In parallel (to cover all bases).
I've outputted everything that leaves the driver and it's telling me exactly what I had in my post above. Yet like you I can't recreate this. Even when using the driver rather than the console.
So I was wondering what other mechanisms could silently affect transactions. And wanted to clear a few questions with you:

The following doesn't seem to throw any errors : if(!graph.tx().isOpen()){graph.tx().open()}; graph.tx().rollback(); graph.tx().commit(). It's a little odd but we've been taking advantage of this assuming that commit() just either ignores it's call or commits an empty mutation. Is this correct or is something else going on? (PS: I will be changing this behavior just to be safe in the event of future changes)
Does the following sequence of separate queries to gremlin-server commit both vertices or does the exception roll back the first? (assuming they're all done in the same session):

if(!graph.tx().isOpen()){graph.tx().open()}
g.addVertex()
g.V().has("uid", "doesNotExists") throws a no element found exception
g.addVertex()
graph.tx().commit()
From the testing I've done this seems to add both vertices but just in case wanted your confirmation.

I would expect the offending transaction's session to eventually timeout, roll the transaction back and free the lock.

Gremlin Server should schedule open transactions to be rolledback when the session times out.  I see the code to do it.  You should see a message in Gremlin Server logs that says something like:
Rolling back any open transactions before killing idle session: {}

I have a test that validates this operation - but I could see where it might not be perfectly validation the situation as you describe it.  I'd have to write a test that used Neo4j (a transactional graph) to nail it. Probably something I'll work on.
Not sure how you kill Gremlin Server, but if it doesn't do an orderly shutdown, then it will rely on the underlying graph to "sort itself out".  I suppose it could be possible that the Gremlin Server thread pools don't shutdown nicely (i.e don't execute the scheduled jobs to kill the sessions), but you would probably know something was fishy as it would take a while to shutdown and sorta hang there (it would probably write something to the log as well to say that a particular pool didn't shutdown properly).
The lock should definitely timeout given: storage.lock.expiry-time  - while looking at that however, i noticed this setting storage.lock.clean-expired which is false by default.  not sure if that matters.
As a side question to @dalaro - why would you want that to be false?  under what circumstance would i want to keep that kinda stuff around?

Does the following sequence of separate queries to gremlin-server commit both vertices or does the exception roll back the first?

I would expect your example to add both vertices.

Ok. I shut titan down via ./bin/titan -v stop so it should be clean. But I never see any session closing messages. I'll try a few more things and run simpler tests to see if I can pinpoint this even further.

But I never see any session closing messages.

dah - i will definitely write that Neo4j test then.....

I'm making some changes to Gremlin Server for 3.0.2 to improve session closing, but I'm not so sure this will do anything much to fix your problem.

Ok great. I'm finding a lot of satellite bugs as I test. All different in some way but they should all be related to the same issue I think.
I've managed to recreate one with a pretty simple workflow. I've done this via the driver in a session Op (sill):

Open session

Create index in query 1:
graphT.tx().rollback();
mgmt=graphT.openManagement();schemaLabel = mgmt.makeVertexLabel("test").make();
TempIndex1 = mgmt.makePropertyKey("uid").dataType(String.class).make();
custIndex = mgmt.buildIndex("keyOnTest",Vertex.class).addKey(TempIndex1).unique().indexOnly(schemaLabel).buildCompositeIndex();
mgmt.setConsistency(custIndex,ConsistencyModifier.LOCK);mgmt.commit();

Start a transaction : if(!graphT.tx().isOpen()){graphT.tx().open()}

Add Vertex : t.addV(label,"test").property("uid","someValue").property("content", "some content")

Commit Transaction : graphT.tx().commit()

Kill thread: This implies that the driver never closes cleanly. This means that the client never manually closes the socket (full-duplex) via stream_socket_shutdown.

We then proceed to running the following which works well (and deletes the vertices):

Open Session
Start a transaction : if(!graphT.tx().isOpen()){graphT.tx().open()}
Delete Vertices : t.V().drop().iterate()
Commit Transaction : graphT.tx().commit()

Once vertices are delete we proceed to recreating the initial vertex:

Open Session
Start a transaction : if(!graphT.tx().isOpen()){graphT.tx().open()}
Add Vertex : t.addV(label,"test").property("uid","someValue").property("content", "some content") > throws an SchemaViolationException (find the full server log bellow)

It looks like the index value isn't properly removed. Not sure why.
43983 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.server.op.OpLoader  - Adding the standard OpProcessor.
43984 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.server.op.OpLoader  - Adding the control OpProcessor.
43988 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.server.op.OpLoader  - Adding the session OpProcessor.
45826 [elasticsearch[Jennifer Kale][clusterService#updateTask][T#1]] INFO  org.elasticsearch.cluster.metadata  - [Jennifer Kale] [titan] create_mapping [contentOnTest]
45889 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.server.op.session.Session  - New session established for ab50d19f-06ae-4ed6-b5d2-5cac87354702
45913 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines  - Loaded nashorn ScriptEngine
45927 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines  - Loaded gremlin-groovy ScriptEngine
46274 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor  - Initialized gremlin-groovy ScriptEngine with scripts/empty-php.groovy
51872 [pool-7-thread-1] INFO  com.thinkaurelius.titan.graphdb.database.management.ManagementLogger  - Received all acknowledgements for eviction [1]
72149 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.server.op.session.Session  - New session established for aac5237b-0186-4baf-bd7f-a97c5229ac4b
72198 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines  - Loaded nashorn ScriptEngine
72243 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines  - Loaded gremlin-groovy ScriptEngine
72624 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor  - Initialized gremlin-groovy ScriptEngine with scripts/empty-php.groovy
72704 [pool-17-thread-1] WARN  com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx  - Query requires iterating over all vertices [()]. For better performance, use indexes
72726 [pool-17-thread-1] WARN  com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx  - Query requires iterating over all vertices [()]. For better performance, use indexes
83943 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.server.op.session.Session  - New session established for b6dbb09b-f301-4f3f-9a20-19c1b0d640c2
83987 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines  - Loaded nashorn ScriptEngine
83998 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines  - Loaded gremlin-groovy ScriptEngine
84350 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor  - Initialized gremlin-groovy ScriptEngine with scripts/empty-php.groovy
85083 [pool-19-thread-1] WARN  org.apache.tinkerpop.gremlin.server.op.AbstractEvalOpProcessor  - Exception processing a script on request [RequestMessage{, requestId=34b6ea95-77dc-47ab-bc08-473ae105e51d, op='eval', processor='session', args={gremlin=t.addV(label,"test").property("uid","someValue").property("content", "some content"), session=b6dbb09b-f301-4f3f-9a20-19c1b0d640c2}}].
com.thinkaurelius.titan.core.SchemaViolationException: Adding this property for key [uid] and value [someValue] violates a uniqueness constraint [keyOnTest]
	at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.addProperty(StandardTitanTx.java:780)
	at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.addProperty(StandardTitanTx.java:706)
	at com.thinkaurelius.titan.graphdb.vertices.AbstractVertex.property(AbstractVertex.java:135)
	at com.thinkaurelius.titan.core.TitanVertex.property(TitanVertex.java:58)
	at com.thinkaurelius.titan.core.TitanVertex.property(TitanVertex.java:19)
	at org.apache.tinkerpop.gremlin.process.traversal.step.sideEffect.AddPropertyStep.sideEffect(AddPropertyStep.java:107)
	at org.apache.tinkerpop.gremlin.process.traversal.step.sideEffect.SideEffectStep.processNextStart(SideEffectStep.java:39)
	at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:140)
	at org.apache.tinkerpop.gremlin.process.traversal.step.util.ExpandableStepIterator.next(ExpandableStepIterator.java:54)
	at org.apache.tinkerpop.gremlin.process.traversal.step.sideEffect.SideEffectStep.processNextStart(SideEffectStep.java:38)
	at org.apache.tinkerpop.gremlin.process.traversal.step.util.AbstractStep.hasNext(AbstractStep.java:140)
	at org.apache.tinkerpop.gremlin.process.traversal.util.DefaultTraversal.hasNext(DefaultTraversal.java:117)
	at org.apache.tinkerpop.gremlin.server.op.AbstractEvalOpProcessor.handleIterator(AbstractEvalOpProcessor.java:201)
	at org.apache.tinkerpop.gremlin.server.op.AbstractEvalOpProcessor.lambda$evalOpInternal$48(AbstractEvalOpProcessor.java:155)
	at org.apache.tinkerpop.gremlin.server.op.AbstractEvalOpProcessor$$Lambda$118/498355686.accept(Unknown Source)
	at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$22(GremlinExecutor.java:255)
	at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor$$Lambda$121/1216096198.call(Unknown Source)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

I'm still digesting this thread.  Thanks for digging.  A couple of things up front:
@spmallette it's not that you want to keep that stuff around (it has no utility).  It's mostly about stability.  clean-expired was bolted onto the lock code after initial implementation, and it hasn't seen as much use as the rest of that code.  clean-expired is also an option that tends to be needed only occasionally (at least, in theory), since reaching the global lock walltime limit automatically fails the holding transaction and is intended to be a rare occurrence in production.  The expired locks don't affect functionality or performance until you accumulate enough of them to annoy the storage backend (e.g. C* wide rows).  Those expired locks are also potentially useful for debugging problems after the fact, and they're more likely to be retained for inspection if they stay in the backend than if they're logged once and blown away.
@PommeVerte regarding "I shut titan down via ./bin/titan -v stop so it should be clean" -- "clean" refers to transaction pools or other runtime state, right? titan.sh stop shouldn't delete the data on disk.

@dalaro yes that is correct. For some reason the locks (or some related mechanism) was outliving titan's execution. stop and then start of titan did not make those elements available for mutations.
I'm currently working my way through what may be some loose ends here before I revisit this issue from a cleaner code base. Will get back to you guys.

Ok.. So after the correction of TINKERPOP3-910 I decided to make a custom build against the titan tp-ci-31 branch with these changes.
I double checked that the 910 patch was correctly in place with my driver test suit and all is good. But I still get the error described in this issue (with berkeleyje, and randomly with cassandra).
I've managed to dumb it down as follows, you will notice this is pretty similar to the 910 issue (it's all in a single channel):

Create unique index (sessionless)
graph.tx().rollback();mgmt=graph.openManagement();
TempIndex1 = mgmt.makePropertyKey('custkey').dataType(String.class).make();
custIndex = mgmt.buildIndex('keyForTest',Vertex.class).addKey(TempIndex1).unique().buildCompositeIndex();
mgmt.setConsistency(custIndex,ConsistencyModifier.LOCK);
mgmt.commit();

Open transaction (in session)
graph.tx().open()

Add vertex (in session)
g.addV().property("name","stephen").property("age", 125).property("custkey","ae1234")

commit (in session)
g.tx().commit()

Read graph (sessionless) <-- this is necessary just as it was for 910
g.V().has("custkey","ae1234").range(0,1)

Open transaction (in session) <-- this works fine thanks to the correction for 910
g.tx().open()

Delete the vertex (in session)
g.V(/*id*/).drop().iterate()

Commit (in session) <-- Error occurs.
g.tx().commit()

I would like to note that no amount of sleep() between operations makes this work. I've played around and I'm quite certain this isn't a case of two quickly executing queries being concurrent.
Also, the error thrown by gremlin-server here is 597 which I feel should probably be 500 instead.
This is actually significant when it comes to client end retry strategies, it doesn't make much sense to retry on a 597.
Is it possible that in a similar fashion to 910 there's a lock that persists (just like the transaction persisted).
Feels like I'm finally narrowing down on this, can't wait to be able to put this in the past as it's the last issue left to resolve for us to set up small client production instance of our software. Fingers crossed.
Here's the full stack from the server:
159096 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.server.op.OpLoader  - Adding the standard OpProcessor.
159098 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.server.op.OpLoader  - Adding the control OpProcessor.
159103 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.server.op.OpLoader  - Adding the session OpProcessor.
160510 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.server.op.session.Session  - New session established for 1f8c8e97-044b-4873-84c8-6b4080473b71
160588 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines  - Loaded nashorn ScriptEngine
160627 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines  - Loaded gremlin-groovy ScriptEngine
160967 [gremlin-server-worker-1] INFO  org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor  - Initialized gremlin-groovy ScriptEngine with scripts/empty-php.groovy
166020 [pool-7-thread-1] INFO  com.thinkaurelius.titan.graphdb.database.management.ManagementLogger  - Received all acknowledgements for eviction [1]
172775 [pool-12-thread-1] ERROR com.thinkaurelius.titan.graphdb.database.StandardTitanGraph  - Could not commit transaction [3] due to storage exception in commit
com.thinkaurelius.titan.core.TitanException: Could not execute operation due to backend exception
	at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:44)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:87)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:141)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:198)
	at com.thinkaurelius.titan.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:118)
	at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:718)
	at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1352)
	at com.thinkaurelius.titan.graphdb.tinkerpop.TitanBlueprintsGraph$GraphTransaction.doCommit(TitanBlueprintsGraph.java:263)
	at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:105)
	at org.apache.tinkerpop.gremlin.structure.Transaction$commit$1.call(Unknown Source)
	at Script4.run(Script4.groovy:1)
	at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:525)
	at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:365)
	at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
	at org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines.eval(ScriptEngines.java:96)
	at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$0(GremlinExecutor.java:258)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: com.thinkaurelius.titan.diskstorage.PermanentBackendException: Permanent failure in storage backend
	at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEKeyValueStore.delete(BerkeleyJEKeyValueStore.java:222)
	at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEStoreManager.mutateMany(BerkeleyJEStoreManager.java:202)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.keyvalue.OrderedKeyValueStoreManagerAdapter.mutateMany(OrderedKeyValueStoreManagerAdapter.java:107)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:90)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:87)
	at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56)
	at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42)
	... 19 more
Caused by: com.sleepycat.je.LockTimeoutException: (JE 5.0.73) Lock expired. Locker 976492442 108_pool-12-thread-1_Txn: waited for lock on database=edgestore LockAddr:2036028358 LSN=0x0/0x1a6d type=WRITE grant=WAIT_PROMOTION timeoutMillis=500 startTime=1445958540756 endTime=1445958541256
Owners: [<LockInfo locker="1213525679 95_gremlin-server-worker-1_Txn" type="READ"/>, <LockInfo locker="976492442 108_pool-12-thread-1_Txn" type="READ"/>]
Waiters: []
Transaction 976492442 108_pool-12-thread-1_Txn owns LockAddr:2036028358 <LockInfo locker="976492442 108_pool-12-thread-1_Txn" type="READ"/>
Transaction 976492442 108_pool-12-thread-1_Txn waits for LockAddr:2036028358

	at com.sleepycat.je.txn.LockManager.newLockTimeoutException(LockManager.java:664)
	at com.sleepycat.je.txn.LockManager.makeTimeoutMsgInternal(LockManager.java:623)
	at com.sleepycat.je.txn.SyncedLockManager.makeTimeoutMsg(SyncedLockManager.java:97)
	at com.sleepycat.je.txn.LockManager.lockInternal(LockManager.java:390)
	at com.sleepycat.je.txn.LockManager.lock(LockManager.java:276)
	at com.sleepycat.je.txn.Txn.lockInternal(Txn.java:498)
	at com.sleepycat.je.txn.Locker.lock(Locker.java:443)
	at com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:2621)
	at com.sleepycat.je.dbi.CursorImpl.delete(CursorImpl.java:1340)
	at com.sleepycat.je.Cursor.deleteNoNotify(Cursor.java:1998)
	at com.sleepycat.je.Database.deleteInternal(Database.java:912)
	at com.sleepycat.je.Database.delete(Database.java:806)
	at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEKeyValueStore.delete(BerkeleyJEKeyValueStore.java:217)
	... 25 more
172778 [pool-12-thread-1] ERROR com.thinkaurelius.titan.graphdb.database.StandardTitanGraph  - Could not commit transaction [3] due to exception
com.thinkaurelius.titan.core.TitanException: Could not execute operation due to backend exception
	at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:44)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:87)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:141)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:198)
	at com.thinkaurelius.titan.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:118)
	at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:718)
	at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1352)
	at com.thinkaurelius.titan.graphdb.tinkerpop.TitanBlueprintsGraph$GraphTransaction.doCommit(TitanBlueprintsGraph.java:263)
	at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:105)
	at org.apache.tinkerpop.gremlin.structure.Transaction$commit$1.call(Unknown Source)
	at Script4.run(Script4.groovy:1)
	at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:525)
	at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:365)
	at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
	at org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines.eval(ScriptEngines.java:96)
	at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$0(GremlinExecutor.java:258)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: com.thinkaurelius.titan.diskstorage.PermanentBackendException: Permanent failure in storage backend
	at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEKeyValueStore.delete(BerkeleyJEKeyValueStore.java:222)
	at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEStoreManager.mutateMany(BerkeleyJEStoreManager.java:202)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.keyvalue.OrderedKeyValueStoreManagerAdapter.mutateMany(OrderedKeyValueStoreManagerAdapter.java:107)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:90)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:87)
	at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56)
	at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42)
	... 19 more
Caused by: com.sleepycat.je.LockTimeoutException: (JE 5.0.73) Lock expired. Locker 976492442 108_pool-12-thread-1_Txn: waited for lock on database=edgestore LockAddr:2036028358 LSN=0x0/0x1a6d type=WRITE grant=WAIT_PROMOTION timeoutMillis=500 startTime=1445958540756 endTime=1445958541256
Owners: [<LockInfo locker="1213525679 95_gremlin-server-worker-1_Txn" type="READ"/>, <LockInfo locker="976492442 108_pool-12-thread-1_Txn" type="READ"/>]
Waiters: []
Transaction 976492442 108_pool-12-thread-1_Txn owns LockAddr:2036028358 <LockInfo locker="976492442 108_pool-12-thread-1_Txn" type="READ"/>
Transaction 976492442 108_pool-12-thread-1_Txn waits for LockAddr:2036028358

	at com.sleepycat.je.txn.LockManager.newLockTimeoutException(LockManager.java:664)
	at com.sleepycat.je.txn.LockManager.makeTimeoutMsgInternal(LockManager.java:623)
	at com.sleepycat.je.txn.SyncedLockManager.makeTimeoutMsg(SyncedLockManager.java:97)
	at com.sleepycat.je.txn.LockManager.lockInternal(LockManager.java:390)
	at com.sleepycat.je.txn.LockManager.lock(LockManager.java:276)
	at com.sleepycat.je.txn.Txn.lockInternal(Txn.java:498)
	at com.sleepycat.je.txn.Locker.lock(Locker.java:443)
	at com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:2621)
	at com.sleepycat.je.dbi.CursorImpl.delete(CursorImpl.java:1340)
	at com.sleepycat.je.Cursor.deleteNoNotify(Cursor.java:1998)
	at com.sleepycat.je.Database.deleteInternal(Database.java:912)
	at com.sleepycat.je.Database.delete(Database.java:806)
	at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEKeyValueStore.delete(BerkeleyJEKeyValueStore.java:217)
	... 25 more
172786 [pool-12-thread-1] WARN  org.apache.tinkerpop.gremlin.server.op.AbstractEvalOpProcessor  - Exception processing a script on request [RequestMessage{, requestId=76a4e8bb-a887-4370-b058-614289020863, op='eval', processor='session', args={gremlin=graphT.tx().commit(), bindings={B_R_ID=4224}, session=1f8c8e97-044b-4873-84c8-6b4080473b71}}].
com.sleepycat.je.LockTimeoutException: (JE 5.0.73) Lock expired. Locker 976492442 108_pool-12-thread-1_Txn: waited for lock on database=edgestore LockAddr:2036028358 LSN=0x0/0x1a6d type=WRITE grant=WAIT_PROMOTION timeoutMillis=500 startTime=1445958540756 endTime=1445958541256
Owners: [<LockInfo locker="1213525679 95_gremlin-server-worker-1_Txn" type="READ"/>, <LockInfo locker="976492442 108_pool-12-thread-1_Txn" type="READ"/>]
Waiters: []
Transaction 976492442 108_pool-12-thread-1_Txn owns LockAddr:2036028358 <LockInfo locker="976492442 108_pool-12-thread-1_Txn" type="READ"/>
Transaction 976492442 108_pool-12-thread-1_Txn waits for LockAddr:2036028358

	at com.sleepycat.je.txn.LockManager.newLockTimeoutException(LockManager.java:664)
	at com.sleepycat.je.txn.LockManager.makeTimeoutMsgInternal(LockManager.java:623)
	at com.sleepycat.je.txn.SyncedLockManager.makeTimeoutMsg(SyncedLockManager.java:97)
	at com.sleepycat.je.txn.LockManager.lockInternal(LockManager.java:390)
	at com.sleepycat.je.txn.LockManager.lock(LockManager.java:276)
	at com.sleepycat.je.txn.Txn.lockInternal(Txn.java:498)
	at com.sleepycat.je.txn.Locker.lock(Locker.java:443)
	at com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:2621)
	at com.sleepycat.je.dbi.CursorImpl.delete(CursorImpl.java:1340)
	at com.sleepycat.je.Cursor.deleteNoNotify(Cursor.java:1998)
	at com.sleepycat.je.Database.deleteInternal(Database.java:912)
	at com.sleepycat.je.Database.delete(Database.java:806)
	at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEKeyValueStore.delete(BerkeleyJEKeyValueStore.java:217)
	at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEStoreManager.mutateMany(BerkeleyJEStoreManager.java:202)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.keyvalue.OrderedKeyValueStoreManagerAdapter.mutateMany(OrderedKeyValueStoreManagerAdapter.java:107)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:90)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:87)
	at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56)
	at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:87)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:141)
	at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:198)
	at com.thinkaurelius.titan.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:118)
	at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:718)
	at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1352)
	at com.thinkaurelius.titan.graphdb.tinkerpop.TitanBlueprintsGraph$GraphTransaction.doCommit(TitanBlueprintsGraph.java:263)
	at org.apache.tinkerpop.gremlin.structure.util.AbstractTransaction.commit(AbstractTransaction.java:105)
	at org.apache.tinkerpop.gremlin.structure.Transaction$commit$1.call(Unknown Source)
	at Script4.run(Script4.groovy:1)
	at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:525)
	at org.apache.tinkerpop.gremlin.groovy.jsr223.GremlinGroovyScriptEngine.eval(GremlinGroovyScriptEngine.java:365)
	at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:233)
	at org.apache.tinkerpop.gremlin.groovy.engine.ScriptEngines.eval(ScriptEngines.java:96)
	at org.apache.tinkerpop.gremlin.groovy.engine.GremlinExecutor.lambda$eval$0(GremlinExecutor.java:258)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
172842 [pool-12-thread-1] INFO  org.apache.tinkerpop.gremlin.server.op.session.Session  - Rolling back open transactions on graphT before killing session: 1f8c8e97-044b-4873-84c8-6b4080473b71

What is your storage.lock.wait-time setting at?  Perhaps you need more time on that?  At this point, I have a hard time explaining what could be wrong because we have it executing nicely on Gremlin Server + Neo4j.  Can't recreate the sequence of failure off of titan.  But we also can't easily recreate it in the console, so......

It's the default (I don't have it set anywhere) 100ms, I changed it to 5000ms but the error still occurs.
Also this is different from what we fixed with gremlin-server and Neo (though it expresses itself in a similar fashion). This is most likely related to the indexing lock system in place setting a lock and not properly closing it (it's composite so at least we don't have to worry about ES?). I'll see tomorrow if I can't provide a branch with a failing test, not really sure where to start with that but at this point I'm desperate to get this fixed.

@PommeVerte do we still have problems here?  I've sorta lost track.

Yes, I'm supposed to provide a failing script at this point, I just haven't had the time to get around to it yet.  I don't have anything planned over the weekend so I'll probably sort it then along with a couple of other things for TP.

@PommeVerte do you plan on coming back to this issue?

yeah. I'm back in the office sorting my priorities. This is definitely in my top 2.

@PommeVerte any updates on this issue?

Just a heads up to let you guys know I'm looking into this again.
 Can't tell just yet if they're related to this issue or not. Will keep you posted later today.

 Once that was switched no more errors.

So far this seems to be fixed!!!!
The failing tests I mentioned earlier in this thread pass with flying colors. I'm no longer getting these Lock warnings etc.. I've tried over cassandra-es, berkley and berkeley-es.

Driver tests PASS
ORM/OGM tests PASS (these used to fail for berkeley)
Application tests Currently running These take a long time to execute so I will post back with more information once they're done. But originally they would end up failing at random taking down all other tests that required DB info. So far it's working correctly.

Will give confirmation tomorrow

Ok so I spoke too soon. I'm still getting these in app. Going to triple check our transaction management to make sur it's not coming from there.
com.sleepycat.je.LockTimeoutException: (JE 5.0.73) Lock expired. Locker 687381160 1355_pool-65-thread-1_Txn: waited for lock on database=edgestore LockAddr:1872487525 LSN=0x0/0x3e2e0 type=WRITE grant=WAIT_PROMOTION timeoutMillis=500 startTime=1458401837583 endTime=1458401838084
Owners: [<LockInfo locker="687381160 1355_pool-65-thread-1_Txn" type="READ"/>, <LockInfo locker="706255862 1351_pool-63-thread-1_Txn" type="READ"/>]
Waiters: []
Transaction 687381160 1355_pool-65-thread-1_Txn owns LockAddr:1872487525 <LockInfo locker="687381160 1355_pool-65-thread-1_Txn" type="READ"/>
Transaction 687381160 1355_pool-65-thread-1_Txn waits for LockAddr:1872487525

Ok so since the OGM/ORM tests passed (as opposed to previously where they systematically failed) I had a suspicion that my app tests were failing because of some awkward transaction management and I was correct. berkeley-es works as expected and all tests pass now. cassandra-es has random es failures (es seems to shutdown) but that's another issue entirely.
I'm going to close this beast of an issue as it seems to be resolved.
Cheers.

This is still an issue for me
I am using titan 1.0 stack within java using berkeleyje-es
with the following fragment
TitanTransaction tx = graph.newTransaction();
GraphTraversalSource g = tx.traversal();
g.V().drop().iterate();
tx.commit();
I do not explicitly drop any edges as they should be implicitly removed.
I repopulate the graph with similar
TitanTransaction tx = graph.newTransaction();
tx.commit();
and on any subsequent call to
TitanTransaction tx = graph.newTransaction();
GraphTraversalSource g = tx.traversal();
g.V().drop().iterate();
tx.commit();
I get the exception
com.sleepycat.je.LockTimeoutException: (JE 5.0.73) Lock expired. Locker 261661398 2303: waited for lock on database=edgestore LockAddr:277627696 LSN=0x0/0x57a8e type=WRITE grant=WAIT_PROMOTION timeoutMillis=500 startTime=1459332076753 endTime=1459332077253
Owners:
at com.sleepycat.je.txn.LockManager.newLockTimeoutException(LockManager.java:664)
at com.sleepycat.je.txn.LockManager.makeTimeoutMsgInternal(LockManager.java:623)
at com.sleepycat.je.txn.SyncedLockManager.makeTimeoutMsg(SyncedLockManager.java:97)
at com.sleepycat.je.txn.LockManager.lockInternal(LockManager.java:390)
at com.sleepycat.je.txn.LockManager.lock(LockManager.java:276)
at com.sleepycat.je.txn.Txn.lockInternal(Txn.java:498)
at com.sleepycat.je.txn.Locker.lock(Locker.java:443)
at com.sleepycat.je.dbi.CursorImpl.lockLN(CursorImpl.java:2621)
at com.sleepycat.je.dbi.CursorImpl.delete(CursorImpl.java:1340)
at com.sleepycat.je.Cursor.deleteNoNotify(Cursor.java:1998)
at com.sleepycat.je.Database.deleteInternal(Database.java:912)
at com.sleepycat.je.Database.delete(Database.java:806)
at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEKeyValueStore.delete(BerkeleyJEKeyValueStore.java:217)
at com.thinkaurelius.titan.diskstorage.berkeleyje.BerkeleyJEStoreManager.mutateMany(BerkeleyJEStoreManager.java:202)
at com.thinkaurelius.titan.diskstorage.keycolumnvalue.keyvalue.OrderedKeyValueStoreManagerAdapter.mutateMany(OrderedKeyValueStoreManagerAdapter.java:107)
at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:90)
at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction$1.call(CacheTransaction.java:87)
at com.thinkaurelius.titan.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:56)
at com.thinkaurelius.titan.diskstorage.util.BackendOperation.execute(BackendOperation.java:42)
at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.persist(CacheTransaction.java:87)
at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.flushInternal(CacheTransaction.java:141)
at com.thinkaurelius.titan.diskstorage.keycolumnvalue.cache.CacheTransaction.commit(CacheTransaction.java:198)
at com.thinkaurelius.titan.diskstorage.BackendTransaction.commitStorage(BackendTransaction.java:118)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.commit(StandardTitanGraph.java:718)
at com.thinkaurelius.titan.graphdb.transaction.StandardTitanTx.commit(StandardTitanTx.java:1352)
at graph.Graph.deleteGraph(Graph.java:83)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
if any progress has been made here and I'm missing something please advise
Cheers

Apologies mybad, found a dangling transaction. All good now

@jimlack Can you share some information on how you found the dangling transaction ? I've been stuck with this problem for a while now. Also let me know if Github comments are not the best way to correspond regarding the issue (please suggest an alternative in that case)
Thanks in advance :)


javaPackages.openjdk17: Fix download URL

The current build attempts to fetch
https://cdn.azul.com/zulu/bin/zulu17.30.19-ca-jdk17.0.1-macosx_aarch64.tar.gz
on aarch64-darwin, and that URL does not exist.
Changing the Zulu version to 17.30.19 to 17.30.15 fixes the URL, and
the hash requires no changes, indicating that the hash must have
originally been computed for version 17.30.15 anyway.

@GrahamcOfBorg build javaPackages.openjdk17

duplicate of #149468?


Add support for numbers in objects

This is necessary for an easier transition from other styles in JS
libraries that let you write styles as objects. All of them support the
following:
const styles = {
  fontSize: 14,
}

This commit adds support for this in styled-components. There is a
handful of properties where appending px doesn't make sense, I stole
the list of non-pixel-properties from JSS (thanks @kof) and simply
append their unit.
Closes #283

Hmm, I forgot unitless numbers. There's things like opacity, which shouldn't get px appended:
const styles = {
  opacity: 0,
}

This is more complex than I initially thought, is it worth it to add this?

88fe81c adds support for unitless numbers.
I'm not 100% if we really want to merge this as it adds quite a bit of code and whitelists, but it'd be a nice addition.

How about moving this out in an extra package like unitless. This way only people moving their codebase to styled-components would have to add this.
I could imagine an API like:
unitless({
  flex: 1,
})

How about moving this out to an extra package. This way only people moving their codebase to styled-components would have to add this.

Hmm, interesting. I hadn't considered that. The thing is, if we want to add a proper API for objects down the line we'll have to have this anyway?

The thing is, if we want to add a proper API for objects down the line we'll have to have this anyway?

Yeah probably even if I don't understand the value of adding the unit automatically. In my opinion appending px or em manually is much more clear.

Yeah.... I don't like this. Lots of CSS properties can have multiple units, and having px as a default is not really correct. Assuming that everything is driven from pixel measurements is a sure fire way to making brittle styles, since you're missing out on that whole part of CSS. The point of Styled Components is to not artificially limit the power of CSS, in fact the very choice of using template strings was to stop this artificial friction, where 14 is a first class JS symbol but 14px or 14pt requires being wrapped in quotes, the same way an object key opacity is fine but font-size needs to be wrapped in a string or camelized.
I get that it's only for objects but it's a design choice (default unit being px) that I don't want to help "standardise" by adding support for it. So it's a no from me.


Allow styling in DateTimePicker component

🐛 Bug Report
There is currently no way to style the DateTimePicker component as a user of it.
To Reproduce

Import DateTimePicker
Try to use className prop
Get compile errors

OR

Import DateTimePicker
Try to use style
Get compile errors

Expected behavior
I should be able to use the style and className props

Hi, I'm Logan Westbrook and I'm looking for a good first issue to get started with so I was hoping to get assigned to this.

I've assigned this to you! Thanks for helping on this!

I'll take a shot at this!

I solved for the compile time errors, but I noticed when testing that when you apply classes or inline styles to the root component (InputGroup from Bootstrap) it only applied some text styling to the DatePicker  subcomponent from the react-datepicker library. Is customizing the calendar the functionality you're after, or was this just about the compiler errors?


Adapt to new yq version in aspect-build/bazel-lib

oci_pushdoes'nt work with the latest aspect-build/bazel-lib because of changes in yq, see https://github.com/aspect-build/bazel-lib/releases/tag/v1.31.0
See also https://github.com/mikefarah/yq/issues/1608

yq has made a breaking change that's not semver compatible. we reverted the change bazel-lib that changed the default version for yq. this should not be necessary anymore.


fix: grant access to minikube/k3d

kwctl is built using rustls enabled. Unfortunately rustls does not support validating IP addresses yet (see https://github.com/kube-rs/kube/issues/1003).
This commit introduces a workaround to address this limitation. This is required to make sure users can run context aware policies against minikube, k3d and other local development solution that deploy kubernetes in this way.

LGTM! Needs the clippy warnings fixed now, though.

doh... I'll handle that and merge it once everything is green


rvm not available to rvm group users

After running test-kitchen for the following recipe, the only user with rvm/rubies available is vagrant.
Only user found in rvm group is 'ubuntu'.  Am I using the correct recipe?
node.default['rvm']['group_users'] = ['ubuntu']

include_recipe 'rvm::system'

You should be using the user recipe


 Daher hab ich den Part erstmal ausgeblendet, damit funktioniert alles wieder problemlos. Der Part sollte zukünftig überarbeitet werden. Danke für den Pull im Voraus.

Thank you for your report!
Can you also tell us how to reproduce this behavior?
Which TYPO3 version?
Which PHP version?
Which error message?
Configuration of your slider?
Content elements?
Images? Or files from a folder? etc. etc.
Best you open an "Issue" for a pull-request at the same time :)
Thank you!

And please report issues in english, so everyone can benefit from it, thanks ;)

Hi Teisi
Can you also tell us how to reproduce this behavior?

Only installed on the current TYPO3 Version

Which TYPO3 version?

11.5.9

Which PHP version?

7.4

Which error message?

I see the following error message on the page tab
https://pastebin.com/WTFSNB78
You can fix it if you uncomment the line 75

Configuration of your slider?

Only linked in TS, nothing special

Content elements?
Images? Or files from a folder? etc. etc.

hmm so I can't reproduce this directly and actually the error message already says the "error":
"Tried resolving a template file for controller action "Standard->index" in format ".html", but none of the paths contained the expected template file (Standard/Index.html). No paths configured."
I think this means it is missing the paths to the HTML template / partial / layout files.
This means either missing TypoScript or PageTs (by default you have to import both with the HauerHeinrich Extensions) or maybe the corresponding parts were overwritten.
Would you like to check this, please? :)
BTW, an out commenting code, is rarely a "bug" fix :)
So if I would like to be able to track that, so we can really fix that without limiting functionalities or something like that. ;)

Strange... I cleaned all temporary files and reEnable the Plugin and now it's gone....
So, we can close this Pull.

Ok. Checked ... Same problem.
I get the same error on a completly different system again.
InvalidTemplateResourceException
Tried resolving a template file for controller action "Standard->hhSlider" in format ".html", but none of the paths contained the expected template file (Standard/HhSlider.html). No paths configured.

Currently, i need to comment the one line to fix it.


@Inherited annotations aren't detected

My annotation classes look like this -

import java.lang.annotation.Inherited

@Target(AnnotationTarget.CLASS)
@Inherited
internal annotation class ScreenshotTest

@Target(AnnotationTarget.CLASS)
internal annotation class EspressoTest

For a test structure like this -

@ScreenshotTest
abstract class BaseScreenshotTest

class ScreenshotTestA : BaseScreenshotTest

abstract class BaseEspressoTest

@EspressoTest
class EspressoTestA : BaseEspressoTest

@EspressoTest
class EspressoTestA : BaseEspressoTest

Using adb shell am instrument -w -e annotation 'com.company.mobile.ScreenshotTest works as expected and picks up ScreenshotTestA.
However, when using dex-parser, the @Test methods in ScreenshotTestA don't have the @ScreenshotTest annotation.
To get around it, instead of only having the annotation on a base class, we have to do away with the @Inherited annotation and manually annotate every class, like in the @EspressoTest example.
It would be amazing if dex parser could also detect and return the @Inherited annotations along with those on the class.

yeah that would be awesome!

We will look into this in more details next quarter, thanks for your patience.

Fixed in https://github.com/linkedin/dex-test-parser/pull/27


Error loading Terraform modules: can't make path relative

In my Gitlab CI pipeline, I was using infracost/infracost:ci-0.10 version and since the last upgrade went in 3 days back .16 my pipeline has been failing with the following error.
$ cd /tmp/base # collapsed multi-line command
time="2023-01-19T10:25:23Z" level=info msg="Evaluating Terraform directory at ."
time="2023-01-19T10:25:23Z" level=info msg="Starting: Downloading Terraform modules"
time="2023-01-19T10:25:23Z" level=info msg="Evaluating Terraform directory at ."
time="2023-01-19T10:25:23Z" level=info msg="Starting: Downloading Terraform modules"
time="2023-01-19T10:25:23Z" level=info msg="Evaluating Terraform directory at ."
time="2023-01-19T10:25:23Z" level=info msg="Starting: Downloading Terraform modules"
time="2023-01-19T10:25:23Z" level=info msg="Evaluating Terraform directory at ."
time="2023-01-19T10:25:23Z" level=info msg="Starting: Downloading Terraform modules"
Error: Error loading Terraform modules: could not load modules for path . Rel: can't make cloudwatch relative to /builds/swateekj/c[36](https://gitlab.com/swateekj/my-infra/-/jobs/3628005383#L36)0/my-infra/

Everytime I retry the pipeline, the module complained about is different (like cloudwatch in the example above).
From the CI file, the command which runs is
cd /tmp/base
infracost breakdown --config-file=${CI_PROJECT_DIR}/${INFRACOST_CONFIG_FILE} \
                    --format=json \
                    --out-file=infracost-base.json

Workaround
For now, I have moved it to the previous infracost/infracost:ci-0.10.15 and it is working fine.

Thanks @swateek for raising this issue.
Some further information would be great to help us debug this issue:

Can you post the config-file you are using to run infracost with. It seems you are using . for multiple projects? Is this because you're using different terraform workspaces/vars?
Could you give us a rough tree structure overview of your project.

Thanks

I am running into this same issue using actions rather than the image itself:
time="2023-01-20T15:40:45Z" level=info msg="Evaluating Terraform directory at infrastructure/terraform" time="2023-01-20T15:40:45Z" level=info msg="Starting: Downloading Terraform modules" time="20[23](https://github.com/mycompany/process-single-template-click/actions/runs/3969010376/jobs/6802923410#step:6:24)-01-20T15:40:45Z" level=info msg="Evaluating Terraform directory at infrastructure/terraform" time="2023-01-20T15:40:46Z" level=info msg="Starting: Downloading Terraform modules" time="2023-01-20T15:40:46Z" level=info msg="Evaluating Terraform directory at infrastructure/terraform" time="2023-01-20T15:40:46Z" level=info msg="Starting: Downloading Terraform modules" Error: Error loading Terraform modules: could not load modules for path infrastructure/terraform Rel: can't make infrastructure/terraform/modules/terraform-aws-elasticache-redis relative to /tmp/
My config file looks like this:
      version: 0.1
      projects:
        - path: infrastructure/terraform
          name: dev
          terraform_var_files:
            - dev.tfvars
          usage_file: ./setup-scripts/variables/infracost-usage-estimates-dev.yml
          env:
            AWS_PROFILE: Terraform-Development

And project structure:
project_root -> infrastructure -> terraform
project_root -> infrastructure -> terraform -> modules -> terraform-aws-elasticache-redis
project_root -> setup-scripts -> variables
Infracost is being installed via:
uses: infracost/actions/setup@v2
I execute infracost in the root directory like this
infracost breakdown --config-file /tmp/infracost.yml --format=json --out-file=/tmp/infracost-base.json

@nick-ppcprotect sorry about that - as a workaround until we get a fix out, you can use the version key in infracost/actions/setup to force the old CLI version to be used:
uses: infracost/actions/setup@v2
with:
  api-key: ${{ secrets.INFRACOST_API_KEY }}
  version: 0.10.15

I was having the same issue - fixing version to 0.10.15 worked for me!

My config file is similar to @nick-ppcprotect, do you still want a peek @hugorut ?

@nick-ppcprotect / @swateek I'm struggling to replicate this issue locally, any additional information you could give would be great.
Here's what I've tried so far. I have a project like so:
infrastructure/
└── terraform
    ├── dev.tfvars
    ├── main.tf
    └── modules
        └── terraform-elasticache-redis
            └── main.tf
infracost.yml

and my infracost.yml file has the following:
version: 0.1
projects:
    - path: infrastructure/terraform
      name: dev
      terraform_var_files:
        - dev.tfvars

running infracost breakdown --config-file infracost.yml gives me a successful output.
Some further questions for you both:

@swateek if you could post your project structure that would be great
@nick-ppcprotect: what does your terraform-elasticache-redis module look like? Does it include any third party modules?

My module is just a local copy of this one: https://github.com/cloudposse/terraform-aws-elasticache-redis.
I do use a relative path when referencing the module if that makes a difference.
eg:
module "redis" {
  source      = "./modules/terraform-aws-elasticache-redis"
  namespace   = var.STAGE
  name        = var.SERVICE
  environment = var.APPLICATION
}

@nick-ppcprotect hmm ok. A further question: do you initialize or use terraform before running infracost?

@hugorut no I don't run any terraform commands.  All I do is source some bash scripts which populate variables.  I then generate tfvar files for each environment based on these which are then used by infracost.

@nick-ppcprotect does this error only happen in your pipeline, or is this something you can replicate locally?

@nick-ppcprotect Ok i've managed to replicate it I think, standby for more info on a fix

Hey @nick-ppcprotect , I've got a potential fix for this in PR but it would be great if you could test it out on your project. Here's the steps to get a build on your machine:

Go here: https://github.com/infracost/infracost/actions/runs/3988483059, and choose an artifact that matches your machine architecture.

Download that and unzip it on your machine. Then you can run the binary inside it just as you would infracost, e.g. : infracost-darwin-arm64 breakdown --path .

If you run into the dreaded can’t be opened because Apple cannot check it for malicious software. error on your machine. You need to follow step 1 outlined here and allow the binary to be run. Let me know if this fixes your issue as we're aiming for a release tomorrow.

Run the build on your project locally just as you did before and report any errors here

@hugorut looks like that fixed it. I was able to run locally without the error.  Thank you!

This is my project structure:
|-load-balancer
 | |-variables.tf
 | |-output.tf
 | |-main.tf
 |-route53
 | |-variables.tf
 | |-output.tf
 | |-main.tf
 |-README.md
 |-ec2
 | |-variables.tf
 | |-output.tf
 | |-main.tf
 | |-provisions
 | | |-docker-compose.yml
 | | |-nginx_config.tftpl
 | | |-env.sample.tftpl
 | | |-bootstrap-machine.sh
 |-.gitlab-ci.yml
 |-variables.tf
 |-output.tf
 |-.pre-commit-config.yaml
 |-cloudwatch
 | |-variables.tf
 | |-main.tf
 |-manage_singlestore.sh
 |-.gitignore
 |-main.tf
 |-config
 | |-amadablam.tfvars
 | |-lhotse.tfvars
 |-infracost.yml

Thanks for reporting this @swateek and the help debugging @nick-ppcprotect, a fix for this issue has been merged into master and will go out with the next release. I'll ping you here when it's live.

had similar issue in github actions, can confirm that writing
steps:
  - name: Setup Infracost
    uses: infracost/actions/setup@v2
    with:
      api-key: ${{ secrets.INFRACOST_API_KEY }}
      version: 0.10.15

fixed my problem!

Hello @swateek , @nick-ppcprotect & @confiq - I can confirm that the fix for this issue has been released as part of the v0.10.17 infracost CLI. Thanks for bringing this to our attention. I'm going to close this issue now, but please comment if you still are experiencing problems.


Automated Package Upgrade

:tada: This PR is included in version 5.0.547 :tada:
The release is available on:

npm package (@latest dist-tag)
GitHub release

Your semantic-release bot :package::rocket:


Integrate TWLGuide into the landing page

Now that we have a central landing page for nintendo custom firmware guides, it might make sense to include a link to the homebrew discord's recommended guide, https://jerbear64.github.io/twlguide/. This has the added benefit of clearing up confusion between whether dsiguide.me or the above guide should be used.

according to plai he's writing his own, issue closed


encoding/asn1: roll back CL 16517?

This CL broke a test at Google in the Go port of keyczar. The test vectors check that signatures with non-minimal integer encodings are still accepted, and after CL 16517 they are not. We need to decide whether to roll back the CL or declare that that usage is not important.

keyczar wants to accept bogus encodings?

/cc @agl

It's not bogus, it's just non-minimal. I have an email thread going with the author.

Okay, please summarize what you find out.
Because I thought ASN.1 was DER and DER was the minimal-form-required version of BER, which I guess isn't the full story.

Could this be related to a bug we have after upgrading from 1.4 to 1.5:
pk error x509: trailing data after ASN.1 of public-key

this ...
http://play.golang.org/p/U6pTR1oC1m
works in 1.4

FYI. Whilst this is changed behaviour from 1.4 - 1.5 it has shown up the fact that one of our systems is generating 'bad' PK's. We can condition the []byte, and fix the culprit, so can move forward. In our case there would be no compelling argument to revert the stricter 1.5
If this is not the same issue then apologies.

@danmux, this issue is about something that changed since 1.5. You may be seeing something in 1.5 different from 1.4 but it's not this. I'd still be interested to hear more about your problem if you'd like to file a new issue. If you can describe the condition you use to detect the issue that might be enough, but of course code is nicer. :-) Thanks.

ok thanks, ftr.. https://github.com/golang/go/issues/13580 and happy for you to delete my polluting comments in here.

After discussion, the plan is to eliminate that keyczar test. The test cared more that the code didn't crash than that it accepted the signature in question. So we do not need to roll back CL 16517.


Alternative function application blocks with vertical currying

For now hidden under the right mouse button, as it needs some more testing and visual tweaks.

The code looks reasonable. As you know or no doubt have guessed, I'm not in favour of the visual changes; I'm not sure if I've written down my motivations anywhere yet, so here goes.

I personally don't know how to parse the visual syntax:

This looks like a function that takes one number (so far so good) and returns one number (eh?). The other input that was curried obviously isn't an input (as it has no anchor attached to it) and because it's on the bottom, but it's also not an output (it's not white-on-black).
I'm worried that, if I don't know how to parse it without instruction, people who are new to Viskell will have an even harder time. However, I do not have any numbers to back this up.
I also don't really get the whole idea of currying, say, the second argument of a four-argument function. I can imagine it being useful in some cases but I've never had to do it in my (admittedly limited) Haskell programming so far. I have used flip once or twice, but that's already in the catalog.
This turns "curry the last three arguments" into three actions (swipes) instead of one.
I'm (usually) stubborn and (occasionally) conservative and LwB-style currying has been in there since June. I'm just starting to like it.

This feature is intentionally hidden under the right mouse button, so that we can compare and gather feedback from other people.
It is visually not finished yet, will look into improving that this weekend.
I think this vertical currying fits better with in the 2D (not linear expression) based structure of viskell.
Making all arguments 'equals' in currying resolves the tricky library design question of argument order.
That it costs more actions in rare cases is a minor issue compared to unstable resizing swiping knots.
If getting used to it is an argument, may I take in account the time thinking about the design before even OWP started :wink:

I don't immediately see us reaching a consensus on this and I wouldn't want to stuff this behind a preference. I'll re-assign to @Flumble.

I'm in favour with everything except for the output label. Like Wander pointed out in https://github.com/wandernauta/viskell/pull/282#issuecomment-160132044 , it is not very clear what the output type of a curried function is, so I'd rather see the whole output type again or have the output type to the right of all the curried arguments (perhaps including an arrow between them).

I would like to see this merged as is, where the vertical currying is only a secondary alternative.
Clearly this needs more work before it is ready as main option, but having a seperate branch is quite inconvenient when many planned changes affect both kind of function blocks.