DRILL-7578: HDF5 Metadata Queries Fail with Large Files

DRILL-7578: HDF5 Metadata Queries Fail with Large Files
Description
HDF5 metadata queries were failing for files with large datasets.  This PR adds a configuration option to avoid projecting the entire dataset in metadata queries and hence solves this issue.
Documentation
Added this to the HDF5 documentation:

projectDatasetInMetadataQuery:  This causes the plugin to project the data set in metadata queries.  Should be false for large files.

Testing
Added new unit test to cover this configuration.

I don't think that it is a good idea to add format config to ignore a specific column for star queries. If the user wants to see specific columns, he should specify them in the query.
By the way, if data is too large, is it possible to read it by batches?

@vvysotskyi Let me give you some context..
This plugin has two ways of interacting with HDF5 files: metadata queries and dataset queries.  HDF5 is like a filesystem within a file, so it can contain many datasets.  The dataset query looks at a specific dataset and projects the columns and rows as you would expect.
Metadata queries are intended to explore the HDF5 itself rather than an individual dataset.  As currently implemented, in metadata queries, the plugin will return the filename, paths, dataset types, from the HDF5 file.  Here's where the problem arose... The metadata query also maps each dataset to a cell in each row.  This is useful because the user gets a preview of the data that is actually in each dataset, however if that dataset is larger than 16MB, Drill crashes.  When I originally implemented this (before EVF) this wasn't an issue because the plugin itself handled pushdown projection, and therefore all the user had to do was exclude the dataset from the query.  However, with EVF it doesn't work that way.
Therefore options are:

Remove this preview functionality entirely
Select some small amount from each dataset and project that in a metadata query
Add a config option to not generate the preview columns in metadata querires.
Convert preview to a string and truncate at size limit.

Of these options, option 3 felt the easiest and most useful to me as it preserved the functionality and gave the users a way to make it work.

@cgivre, thanks for the explanation. The easiest way is not always the best way.
In this case, I think we should add support for the projection pushdown. As an example, you may use the Avro format plugin, it also uses EVF, but supports projection pushdown.

@vvysotskyi
I was under the impression that the EVF handled pushdown project automatically.  @paul-rogers can you comment?
Regardless, if I was to add pushdown projection to this plugin, it would still cause an issue in that SELECT *  metadata queries would still fail for large files.  Whereas I think there would be a better user experience having the config option which, when enabled, would make SELECT * queries work.

@cgivre, in your previous comment, you have mentioned that when plugin itself handled projection pushdown, this issue wasn't observed, so I assumed that it wasn't done by EVF (at least in this case).
Regarding user experience, I doubt that it would be better for user experience to introduce this config option, it just confuses the user: SELECT * should return all the records, but it will ignore specific column. What behavior is expected if the user would specify this column explicitly, or if he would add a filter with this column, or even aggregation?

@vvysotskyi
Thanks for your responses and feedback.   Maybe it would help if you saw how this worked:
apache drill> select *
2..semicolon> from dfs.test.`dset.h5`;
+-------+-----------+-----------+--------------------------------------------------------------------------+
| path  | data_type | file_name |                                 int_data                                 |
+-------+-----------+-----------+--------------------------------------------------------------------------+
| /dset | DATASET   | dset.h5   | [[1,2,3,4,5,6],[7,8,9,10,11,12],[13,14,15,16,17,18],[19,20,21,22,23,24]] |
+-------+-----------+-----------+--------------------------------------------------------------------------+

The column that is in question is the int_data column.  What's happening is if that column is greater than 16MB, Drill runs out of memory.  If a user is actually interested in analyzing that data, they should be using a data query which looks like this:
apache drill> SELECT * FROM table(dfs.test.`dset.h5` (type => 'hdf5', defaultPath => '/dset'));
+-----------+-----------+-----------+-----------+-----------+-----------+
| int_col_0 | int_col_1 | int_col_2 | int_col_3 | int_col_4 | int_col_5 |
+-----------+-----------+-----------+-----------+-----------+-----------+
| 1         | 2         | 3         | 4         | 5         | 6         |
| 7         | 8         | 9         | 10        | 11        | 12        |
| 13        | 14        | 15        | 16        | 17        | 18        |
| 19        | 20        | 21        | 22        | 23        | 24        |
+-----------+-----------+-----------+-----------+-----------+-----------+
4 rows selected (0.172 seconds)

All that is happening in this PR is changing the default behavior in the metadata view.  I set the default to false so that a user would explicitly have to enable this capability.
If you're not liking this option, what would you suggest as an alternative.  I don't think the current situation is ideal where Drill runs out of memory and crashes...

A bit confused by the crash on 16MB part. The problem description is vague. Is there a stack trace somewhere?
EVF is designed to limit individual vectors to 16MB. Once you hit that size, EVF does an "overflow" move: it copies the last record (the one that does not fit) into a new batch, then tell you to return the now-full batch.
If you are seeing a crash, it could be that there is a bit in the overflow logic. (That logic is quite complex.) The proper fix, then, would be for me to find and fix that bug.
Regarding projection: yes, EVF handles projection. You can ask for writers for all your columns, EVF gives you a "dummy" writer for those that are not projected. While top-level columns can be handled by a plugin easily (just set some flags, say), nested columns are very hard to implement in the plugin. EVF provides a uniform way to handle projection at all levels. And, for top level arrays such as column, EVF also handles per-element projection.
As a result, the only difference between EVF-based projection and roll-you-own is that, with EVF, the easiest path is to read the data, give it to the column writer, and let the column writer throw it away. Works well for sequential formats such as JSON and CSV.
If your format is random-access (you have to request each column, as in Parquet), then it is better to ask if the column is projected. But, if your data structure is nested, you have to do this at each level.
So, with that explanation out of the way, what about EVF projection is not working the way roll-your-own did? Let's figure that out and fix it.

@paul-rogers
Here's what's happening:
apache drill> select *
2..semicolon> from dfs.test.`eFitOut.h5`;
Error: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.

A single column value is larger than the maximum allowed size of 16 MB
Fragment 0:0

[Error Id: 1d3f37ca-e7e3-48f5-9c7d-cb24bd494a67 on localhost:31010] (state=,code=0)

When I dug into this, I found that it was one of the dataset columns that has a single-cell value greater than 16GB.  This PR basically disables the reader from attempting to retrieve the datasets and then we avoid the whole issue.
What made me do this also is even if you only select other columns, you still get the error:
apache drill> select path, data_type
2..semicolon> from dfs.test.`eFitOut.h5`;
Error: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.

A single column value is larger than the maximum allowed size of 16 MB
Fragment 0:0

[Error Id: 1af14cb0-9bce-488a-9d2d-aca5736670e3 on localhost:31010] (state=,code=0)
apache drill>

So to conclude:

There may be a bug in the EVF projection with large fields.  (I don't know...)
This PR fixes the issue for HDF5 by removing the datasets from the metadata view.

I should note that when the datasets are not projected in the metadata view, the queries execute without issues.

Here are the stack traces:
apache drill> select *
2..semicolon> from dfs.test.`eFitOut.h5`;
Error: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.

A single column value is larger than the maximum allowed size of 16 MB
Fragment 0:0

[Error Id: 8f722576-9dfc-4462-8acd-dcba76815f37 on localhost:31010] (state=,code=0)
java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.

A single column value is larger than the maximum allowed size of 16 MB
Fragment 0:0

[Error Id: 8f722576-9dfc-4462-8acd-dcba76815f37 on localhost:31010]

	at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:537)
	at org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:609)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1278)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:58)
	at org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:667)
	at org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1102)
	at org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1113)
	at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
	at org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:200)
	at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
	at org.apache.calcite.avatica.AvaticaStatement.execute(AvaticaStatement.java:217)
	at sqlline.Commands.executeSingleQuery(Commands.java:1054)
	at sqlline.Commands.execute(Commands.java:1003)
	at sqlline.Commands.sql(Commands.java:967)
	at sqlline.SqlLine.dispatch(SqlLine.java:734)
	at sqlline.SqlLine.begin(SqlLine.java:541)
	at sqlline.SqlLine.start(SqlLine.java:267)
	at sqlline.SqlLine.main(SqlLine.java:206)
Caused by: org.apache.drill.common.exceptions.UserRemoteException: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.

A single column value is larger than the maximum allowed size of 16 MB
Fragment 0:0

[Error Id: 8f722576-9dfc-4462-8acd-dcba76815f37 on localhost:31010]

	at org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:125)
	at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
	at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
	at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
	at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Exception: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.

A single column value is larger than the maximum allowed size of 16 MB
Fragment 0:0

[Error Id: 8f722576-9dfc-4462-8acd-dcba76815f37 on localhost:31010]
	at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:653)
	at org.apache.drill.exec.physical.resultSet.impl.ResultSetLoaderImpl.overflowed(ResultSetLoaderImpl.java:650)
	at org.apache.drill.exec.physical.resultSet.impl.ColumnState$PrimitiveColumnState.overflowed(ColumnState.java:73)
	at org.apache.drill.exec.vector.accessor.writer.BaseScalarWriter.overflowed(BaseScalarWriter.java:222)
	at org.apache.drill.exec.vector.accessor.writer.AbstractFixedWidthWriter.resize(AbstractFixedWidthWriter.java:251)
	at org.apache.drill.exec.vector.accessor.writer.AbstractFixedWidthWriter$BaseFixedWidthWriter.prepareWrite(AbstractFixedWidthWriter.java:99)
	at org.apache.drill.exec.vector.accessor.writer.AbstractFixedWidthWriter$BaseFixedWidthWriter.prepareWrite(AbstractFixedWidthWriter.java:86)
	at org.apache.drill.exec.vector.accessor.ColumnAccessors$Float8ColumnWriter.setDouble(ColumnAccessors.java:1150)
	at org.apache.drill.exec.store.hdf5.HDF5BatchReader.doubleMatrixHelper(HDF5BatchReader.java:839)
	at org.apache.drill.exec.store.hdf5.HDF5BatchReader.mapDoubleMatrixField(HDF5BatchReader.java:812)
	at org.apache.drill.exec.store.hdf5.HDF5BatchReader.projectDataset(HDF5BatchReader.java:546)
	at org.apache.drill.exec.store.hdf5.HDF5BatchReader.projectMetadataRow(HDF5BatchReader.java:392)
	at org.apache.drill.exec.store.hdf5.HDF5BatchReader.next(HDF5BatchReader.java:340)
	at org.apache.drill.exec.physical.impl.scan.framework.ShimBatchReader.next(ShimBatchReader.java:132)
	at org.apache.drill.exec.physical.impl.scan.ReaderState.readBatch(ReaderState.java:414)
	at org.apache.drill.exec.physical.impl.scan.ReaderState.next(ReaderState.java:371)
	at org.apache.drill.exec.physical.impl.scan.ScanOperatorExec.nextAction(ScanOperatorExec.java:263)
	at org.apache.drill.exec.physical.impl.scan.ScanOperatorExec.next(ScanOperatorExec.java:234)
	at org.apache.drill.exec.physical.impl.protocol.OperatorDriver.doNext(OperatorDriver.java:201)
	at org.apache.drill.exec.physical.impl.protocol.OperatorDriver.start(OperatorDriver.java:179)
	at org.apache.drill.exec.physical.impl.protocol.OperatorDriver.next(OperatorDriver.java:129)
	at org.apache.drill.exec.physical.impl.protocol.OperatorRecordBatch.next(OperatorRecordBatch.java:150)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:122)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:114)
	at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:64)
	at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:87)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:177)
	at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104)
	at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:83)
	at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94)
	at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:326)
	at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:313)
	at .......(:0)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
	at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:313)
	at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
	at .......(:0)

This query should not fail:
apache drill> select path, data_type
2..semicolon> from dfs.test.`eFitOut.h5`;
Error: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.

A single column value is larger than the maximum allowed size of 16 MB
Fragment 0:0

[Error Id: 28070678-b89c-444d-9f0c-cbd3fb69c431 on localhost:31010] (state=,code=0)
java.sql.SQLException: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.

A single column value is larger than the maximum allowed size of 16 MB
Fragment 0:0

[Error Id: 28070678-b89c-444d-9f0c-cbd3fb69c431 on localhost:31010]

	at org.apache.drill.jdbc.impl.DrillCursor.nextRowInternally(DrillCursor.java:537)
	at org.apache.drill.jdbc.impl.DrillCursor.loadInitialSchema(DrillCursor.java:609)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:1278)
	at org.apache.drill.jdbc.impl.DrillResultSetImpl.execute(DrillResultSetImpl.java:58)
	at org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:667)
	at org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1102)
	at org.apache.drill.jdbc.impl.DrillMetaImpl.prepareAndExecute(DrillMetaImpl.java:1113)
	at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
	at org.apache.drill.jdbc.impl.DrillConnectionImpl.prepareAndExecuteInternal(DrillConnectionImpl.java:200)
	at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
	at org.apache.calcite.avatica.AvaticaStatement.execute(AvaticaStatement.java:217)
	at sqlline.Commands.executeSingleQuery(Commands.java:1054)
	at sqlline.Commands.execute(Commands.java:1003)
	at sqlline.Commands.sql(Commands.java:967)
	at sqlline.SqlLine.dispatch(SqlLine.java:734)
	at sqlline.SqlLine.begin(SqlLine.java:541)
	at sqlline.SqlLine.start(SqlLine.java:267)
	at sqlline.SqlLine.main(SqlLine.java:206)
Caused by: org.apache.drill.common.exceptions.UserRemoteException: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.

A single column value is larger than the maximum allowed size of 16 MB
Fragment 0:0

[Error Id: 28070678-b89c-444d-9f0c-cbd3fb69c431 on localhost:31010]

	at org.apache.drill.exec.rpc.user.QueryResultHandler.resultArrived(QueryResultHandler.java:125)
	at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:422)
	at org.apache.drill.exec.rpc.user.UserClient.handle(UserClient.java:96)
	at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:273)
	at org.apache.drill.exec.rpc.RpcBus$InboundHandler.decode(RpcBus.java:243)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:88)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:312)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:286)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:86)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:335)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1294)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:356)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:342)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:911)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:131)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:580)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:497)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)
	at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:131)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Exception: RESOURCE ERROR: One or more nodes ran out of memory while executing the query.

A single column value is larger than the maximum allowed size of 16 MB
Fragment 0:0

[Error Id: 28070678-b89c-444d-9f0c-cbd3fb69c431 on localhost:31010]
	at org.apache.drill.common.exceptions.UserException$Builder.build(UserException.java:653)
	at org.apache.drill.exec.physical.resultSet.impl.ResultSetLoaderImpl.overflowed(ResultSetLoaderImpl.java:650)
	at org.apache.drill.exec.physical.resultSet.impl.ColumnState$PrimitiveColumnState.overflowed(ColumnState.java:73)
	at org.apache.drill.exec.vector.accessor.writer.BaseScalarWriter.overflowed(BaseScalarWriter.java:222)
	at org.apache.drill.exec.vector.accessor.writer.AbstractFixedWidthWriter.resize(AbstractFixedWidthWriter.java:251)
	at org.apache.drill.exec.vector.accessor.writer.AbstractFixedWidthWriter$BaseFixedWidthWriter.prepareWrite(AbstractFixedWidthWriter.java:99)
	at org.apache.drill.exec.vector.accessor.writer.AbstractFixedWidthWriter$BaseFixedWidthWriter.prepareWrite(AbstractFixedWidthWriter.java:86)
	at org.apache.drill.exec.vector.accessor.ColumnAccessors$Float8ColumnWriter.setDouble(ColumnAccessors.java:1150)
	at org.apache.drill.exec.store.hdf5.HDF5BatchReader.doubleMatrixHelper(HDF5BatchReader.java:839)
	at org.apache.drill.exec.store.hdf5.HDF5BatchReader.mapDoubleMatrixField(HDF5BatchReader.java:812)
	at org.apache.drill.exec.store.hdf5.HDF5BatchReader.projectDataset(HDF5BatchReader.java:546)
	at org.apache.drill.exec.store.hdf5.HDF5BatchReader.projectMetadataRow(HDF5BatchReader.java:392)
	at org.apache.drill.exec.store.hdf5.HDF5BatchReader.next(HDF5BatchReader.java:340)
	at org.apache.drill.exec.physical.impl.scan.framework.ShimBatchReader.next(ShimBatchReader.java:132)
	at org.apache.drill.exec.physical.impl.scan.ReaderState.readBatch(ReaderState.java:414)
	at org.apache.drill.exec.physical.impl.scan.ReaderState.next(ReaderState.java:371)
	at org.apache.drill.exec.physical.impl.scan.ScanOperatorExec.nextAction(ScanOperatorExec.java:263)
	at org.apache.drill.exec.physical.impl.scan.ScanOperatorExec.next(ScanOperatorExec.java:234)
	at org.apache.drill.exec.physical.impl.protocol.OperatorDriver.doNext(OperatorDriver.java:201)
	at org.apache.drill.exec.physical.impl.protocol.OperatorDriver.start(OperatorDriver.java:179)
	at org.apache.drill.exec.physical.impl.protocol.OperatorDriver.next(OperatorDriver.java:129)
	at org.apache.drill.exec.physical.impl.protocol.OperatorRecordBatch.next(OperatorRecordBatch.java:150)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:122)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:114)
	at org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext(AbstractUnaryRecordBatch.java:64)
	at org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext(ProjectRecordBatch.java:87)
	at org.apache.drill.exec.record.AbstractRecordBatch.next(AbstractRecordBatch.java:177)
	at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:104)
	at org.apache.drill.exec.physical.impl.ScreenCreator$ScreenRoot.innerNext(ScreenCreator.java:83)
	at org.apache.drill.exec.physical.impl.BaseRootExec.next(BaseRootExec.java:94)
	at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:326)
	at org.apache.drill.exec.work.fragment.FragmentExecutor$1.run(FragmentExecutor.java:313)
	at .......(:0)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1730)
	at org.apache.drill.exec.work.fragment.FragmentExecutor.run(FragmentExecutor.java:313)
	at org.apache.drill.common.SelfCleaningRunnable.run(SelfCleaningRunnable.java:38)
	at .......(:0)

@cgivre, thanks for the stack traces. One of the annoying aspects of Drill errors are the first two chunks of traces: they will always be the same as they are on the client side.
The 16MB error is basically saying that we are writing more than 16MB for a single column in a single row. This is generally Not a Good Thing.
I guess I'm surprised that you are able to get that error, however. You said you are writing INT values. A batch can have at most 32K values. 32K * 4 = 128K. If you have BIGINT, it is 256K. The same is true if you write a DOUBLE.
Looks like your code may be writing an array given the "doubleMatrixHelper" name. If so, then you can get to 16 MB if you write a square matrix with more than sqrt(16M/8) ~= 1400 elements. Are you?
The problem with such large arrays are several. First, clients can't consume them; they have to be flattened. You will get 16M / 8 = 2M rows as a result. Second, allocating buffers larger than 16 MB will fragment memory.
We can provide an option to disable the 16MB limit. But, this is using a table saw without the guard: it will work some times, will cause mayhem other times. Does not help with the other issue: that an xDBC client can't really consume that volume of data.
What is the use case here to a) show such large volumes of data in a schema view, and b) retrieve that volume of data even in a data query?

@paul-rogers
I don't really think it's useful to return that much data and that was why I added the config option to disable it.  I do think having a data preview can be useful but when you get a large dataset, the utility diminishes pretty quickly.

As it turns out, you'll get the same error if you try to do a normal query on the data. So, we need a rule for what to do if a column is too large even at runtime. You can fail the query (as now), you can truncate the data, you can split the data into multiple rows.
Here is where it would be super-helpful if Drill supported normal SQL warnings: then we could truncate the data when it is optional (as here), and issue a warning. Sigh.
Working around a data size limitation is not a good use for a plugin config option. The user would need a different config for each file.
Can you, instead, support some special column that is the preview? Don't include the preview column in SELECT *. (I think there is an EVF trick for that, I'll have to refresh my memory. I think I added it for one of your other plugins.) If the user wants the preview, they can ask SELECT name, preview FROM ....
Otherwise, the error is correct: the data is too large for a Drill column.

@paul-rogers , @vvysotskyi
I was looking at the HDF5 library and it turns out that there is metadata available for the datasets.  Therefore I think I'm going to do three things after removing the config option:

Add new columns to the metadata view with the dataset information (size, data type, dimensions)
Check the data size prior to projection and truncate it if it is too big.
Update the documentation.

Does this sound like a good plan?

@cgivre, sounds like a good plan.
By the way, it occurred to me that the original "preview" idea may have another issue. Drill is SQL-based; clients work best when a column has a specific type. On the other hand, if HDF5 is a file system, and we want to preview files, each file may have a different kind of data: records in one, strings in another, a matrix in a third. If we try to write each of these into a "preview" column, not only do we have a size issue, we also have a type issue: all of these examples are different types.
On your laptop, the OS gives you a preview of each file. The common denominator is the graphic tile which might be a tiny version of an image or a video frame, might be an app icon, or whatever. Point is, the OS converts all the many file types into a common format: the preview tile.
If the "metadata" view scans all files, the preview can be huge (the entire HDF5 content, perhaps.) A preview should be small. Rendering a tile may not be super helpful. But, perhaps a brief text representation. For your bit column, maybe: "[[123456.78, 98745.43, ...".
Your proposed solution of using HDF5-provided metadata is also good, it makes your metadata query more like an "ls -l" equivalent.

@paul-rogers @vvysotskyi
See above comment.  I removed the config option and added logger warnings if the data is truncated.  Again, this is just for "preview" mode so real data queries are not affected.
In doing this PR, I discovered that the HDF5 format allows for arrays within compound fields.
This functionality is not supported by Drill so I added a warning for that.  In the future, or if anyone asks for it, I may add it but for now, I'm leaving that alone.

@cgivre, one more design-level comment about this particular file format. You've mentioned several times that HDF5 is "a file system within a file." It finally clicked: we need need to treat this file as a directory, not a file. This means adding a layer of schema in Calcite planning:
SELECT * FROM `dfs`.`some/path/myFile.hdf5`.`dataSet1`

This would let the reader load only data from dataSet1, using only the schema from that data set.
(Can't use slashes; that is a notation for the Hadoop file system.)
Fortunately, Calcite seems to allow any number of schema levels. It is why we can have plugins, workspaces, etc. The challenge is to provide some way for a format plugin to influence the planner and say, "hey, if you do a query against me, ask me to resolve all path elements below my file name."
Again, not something for this PR. But, it is something we can think about as we try to improve our storage plugin API.

@paul-rogers
I think this is ready for the second pass at review.  Thx

Nit: toStringmethod will be called anyway, so no need to indicate it.
@arina-ielchiieva
Which line are you referring to?

@arina-ielchiieva Can we merge this PR now?  Thx

@cgivre don't see that @paul-rogers approval, so we should wait for it first.

@paul-rogers Do you approve of this PR?

@cgivre please squash the commits and rebase on the latest master. Thanks!

@arina-ielchiieva @paul-rogers
Thanks for the review!
Rebased and squashed.


Don't use interpolation inside a gettext string

Same fix for fine branch will come in a separate PR.
@miq-bot add_label fine/no

Backported to Fine via https://github.com/ManageIQ/manageiq/pull/14759


[Akamai]

Please follow the guide below

You will be asked some questions and requested to provide some information, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your issue (like this: [x])
Use the Preview tab to see what your issue will actually look like

Make sure you are using the latest version: run youtube-dl --version and ensure your version is 2018.07.10. If it's not, read this FAQ entry and update. Issues with outdated version will be rejected.

[x] I've verified and I assure that I'm running youtube-dl 2018.07.10

Before submitting an issue make sure you have:

[x] At least skimmed through the README, most notably the FAQ and BUGS sections
[x] Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

[ ] Bug report (encountered problems with youtube-dl)
[x] Site support request (request for adding support for a new site)
[ ] Feature request (request for a new functionality)
[ ] Question
[ ] Other

The following sections concretize particular purposed issues, you can erase any section (the contents between triple ---) not applicable to your issue

If the purpose of this issue is a bug report, site support request or you are not completely sure provide the full verbose output as follows:
Add the -v flag to your command line you run youtube-dl with (youtube-dl -v <your command line>), copy the whole output and insert it here. It should look similar to one below (replace it with your log inserted between triple ```):
youtube-dl -o "C:/Video/%(title)s.%(ext)s" "http://tvegolf-i.akamaihd.net/hls/live/218225/golfx/1896k/prog.m3u8" --no-check-certificate --verbose --ignore-errors
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-o', 'C:/Video/%(title)s.%(ext)s', 'http://tvegolf-i.akamaihd.net/hls/live/218225/golfx/1896k/prog.m3u8', '--no-check-certificate', '--verbose', '--ignore-errors']
[debug] Encodings: locale cp1252, fs mbcs, out cp850, pref cp1252
[debug] youtube-dl version 2018.07.10
[debug] Python version 3.4.4 (CPython) - Windows-10-10.0.15063
[debug] exe versions: ffmpeg 2.2.3, ffprobe 2.2.3
[debug] Proxy map: {}
[generic] prog: Requesting header
[generic] prog: Downloading m3u8 information
ERROR: Failed to download m3u8 information: HTTP Error 403: Forbidden (caused by HTTPError()); please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp4etivjk5\build\youtube_dl\extractor\common.py", line 599, in _request_webpage
  File "C:\Users\dst\AppData\Roaming\Build archive\youtube-dl\rg3\tmp4etivjk5\build\youtube_dl\YoutubeDL.py", line 2211, in urlopen
  File "C:\Python\Python34\lib\urllib\request.py", line 470, in open
  File "C:\Python\Python34\lib\urllib\request.py", line 580, in http_response
  File "C:\Python\Python34\lib\urllib\request.py", line 508, in error
  File "C:\Python\Python34\lib\urllib\request.py", line 442, in _call_chain
  File "C:\Python\Python34\lib\urllib\request.py", line 588, in http_error_default
...
<end of log>

If the purpose of this issue is a site support request please provide all kinds of example URLs support for which should be included (replace following example URLs by yours):

Official stream: http://tvegolf-i.akamaihd.net/hls/live/218225/golfx/1896k/prog.m3u8

Note that youtube-dl does not support sites dedicated to copyright infringement. In order for site support request to be accepted all provided example URLs should not violate any copyrights.

Description of your issue, suggested solution and other information
Please add support for the following URL -
http://tvegolf-i.akamaihd.net/hls/live/218225/golfx/1896k/prog.m3u8
The method is probably essentially the same as that used for nbc.com. You just need to be able to pass a cookie such as alid=6e1U8ZkkIs7nhu1zmoRFnA== to ffmpeg. User-agent header may also be required.

It's your responsibility to pass cookies, headers and mimic browser's behavior when you use non source URLs.


Color Name of ColorSetting in Report State

Hi,
When sending the report state for colorsetting trait, we send both spectrumRGB and name of the color. But we check the homegraph, only spectrumRGB is set, there is no color name even we submit that.
But from Nick's Medium blog, I do see there was color name in homegraph.
Cheers.

Can you provide the payload that you're sending, as confirmation?

Sure. We use gRPC. The report sent was:
Report state request:
request_id:"e044c417-f3fb-40ef-8762-131c9b67cb9e"
agent_user_id:"xxx"
payload:<
devices:<
states:<
fields:<key:"lamps38" value:<
struct_value:<
fields:<key:"brightness" value:<number_value:100 > >
fields:<key:"color" value:<
struct_value:<
fields:<key:"name" value:<string_value:"Blue" > >
fields:<key:"spectrumRGB" value:<number_value:255 > >
> > >
fields:<key:"on" value:<bool_value:true > >
fields:<key:"online" value:<bool_value:true > >
> > >
>
>

Report state result:
request_id:"e044c417-f3fb-40ef-8762-131c9b67cb9e"

Going thru the back of issues, this sounds like an general issue with the Smart Home platform, and not something specific to the sample code hosted in the repo.
I order to give more visibility, I would suggest following the instructions about filing issues in the public issue tracker at https://developers.google.com/assistant/smarthome/support.

Going thru the back of issues, this sounds like an general issue with the Smart Home platform, and not something specific to the sample code hosted in the repo.
I order to give more visibility, I would suggest following the instructions about filing issues in the public issue tracker at https://developers.google.com/assistant/smarthome/support.


Remove wrapper zip after unzipping

Expected Behavior

To save space, it would be nice, if the wrapper would remove the downloaded zip file after it has been unzipped.
Current Behavior

Unused zip files are consuming disk space in ~/.gradle/wrapper/dists, e.g.:
gradle-4.1/
gradle-4.1-bin.zip
gradle-4.1-bin.zip.lck
gradle-4.1-bin.zip.ok

Still a great feature request.

It would be nice if the zips were removed, especially for CI agents where users don't know zip is never really re-used.

I've been caching the zip file in CI:

it's quicker to cache than the exploded archive
I don't have to download it from Gradle
Not a fan of this change.


Implement Send for OneshotBody

Currently it's not somehow...

The problem is actually with the PrivateBody trait. Maybe with the lifetimes in there, not sure, maybe a limit to the (unstable) generic_associated_types feature?


Expand terraform to work across the matrix of distributions.

The testing for Chef-Infra-Server before release should be run across all the supported distributions.

Redhat Enterprise Linux 6
Redhat Enterprise Linux 7
~SUSE Linux Enterprise Server 11~ (no longer supported)
SUSE Linux Enterprise Server 12
Ubuntu 16.04
Ubuntu 18.04

We currently have 2 scenarios working for testing on an adhoc basis:

standalone upgrade
tiered fresh install ipv6

To Test:
If we launch the same scenario with different versions of artifact to be tested, it creates multiple instances in ec2.
Would running make destroy for a scenario, destroy then all?

@PrajaktaPurohit,
I've opened PR #1791 to enable concurrent testing across the matrix of distributions as well as IPv4 vs IPv6.
In reviewing this issue I'm seeing your "To Test" section describing a situation where someone would be testing the same scenario multiple times concurrently specifying different artifact versions.  This seems like an odd situation because I would have expected the focus of testing to be on an artifact related to the current git working tree.  If this is a realistic scenario, I'll have to further expand the workspace naming to accommodate both versions to ensure uniqueness.

Closing this since we now support running a version on multiple distributions. (#1791) Will create another issue to track being able to run multiple upgrade versions on multiple distributions.


Figure out a better way to associate Rec with Rd

Currently,
pub struct Rec {
    pub rd: &'static Rd,
    vcpuid: usize,
    ripas: Ripas,
}

One Rd can have multiple Recs. So, roughly thinking, we need to have the ability to manage the reference count of the rd. (need to look at RMM spec and TF-RMM, and figure out what would be the best way to associate Rec with Rd)

More generally speaking, what this issue wants to solve is

if X is in use, Y must not be freed.

For example, there are multiple RECs in use, the associated RD must not be destroyed even when RMI_REALM_DESTROY called. (error code: RMI_ERROR_IN_USE)
The cases in this category are:

RD (REALM_DESTROY):  If there are other instances (e.g., Rec) that points to an RD, the RD should not be destroyed.
REC (REC_DESTROY):  If there are other instances (e.g., Run) that points an REC, the REC should not be destroyed.

More concrete scenario:
(1) Create Rd
(2) Create Rec
(3) Destroy Rd --> Check if this Rd's reference count is 1 by (2).
Possible approaches to do:
(1)


Some HLLD tests failing on H2P but not C-3PO

@helenarichie brought it to my attention in PR #257 that the following three HLLD tests are failing on H2P but not C-3PO. The errors are small and likely not an issue but I'll sort it out soon.

tMHDHlldInternalComputeStarState.CorrectInputNonDegenerateExpectCorrectOutput
tMHDHlldInternalStarFluxes.CorrectInputNonDegenerateExpectCorrectOutput
tMHDHlldInternalDoubleStarFluxes.CorrectInputExpectCorrectOutput

If it's helpful, I saved the output from when I ran them the other day. I'll post them here in case it is:
[ RUN      ] tMHDHlldInternalDoubleStarFluxes.CorrectInputExpectCorrectOutput
src/utils/../utils/testing_utilities.h:149: Failure
Value of: areEqual
  Actual: false
Expected: true
Difference in Case 2, DensityStarFlux
The fiducial value is:       10.040447333773216
The test value is:           10.040447333773258
The absolute difference is:  4.2632564145606011e-14
The ULP difference is:       24

src/utils/../utils/testing_utilities.h:149: Failure
Value of: areEqual
  Actual: false
Expected: true
Difference in Case 2, EnergyStarFlux
The fiducial value is:       59.061079503595323
The test value is:           59.061079503595266
The absolute difference is:  5.6843418860808015e-14
The ULP difference is:       8

[  FAILED  ] tMHDHlldInternalDoubleStarFluxes.CorrectInputExpectCorrectOutput (0 ms)

[ RUN      ] tMHDHlldInternalStarFluxes.CorrectInputNonDegenerateExpectCorrectOutput
src/utils/../utils/testing_utilities.h:149: Failure
Value of: areEqual
  Actual: false
Expected: true
Difference in Case 1, EnergyStarFlux
The fiducial value is:       -21413.063415617493
The test value is:           -21413.063415617515
The absolute difference is:  2.1827872842550278e-11
The ULP difference is:       6

src/utils/../utils/testing_utilities.h:149: Failure
Value of: areEqual
  Actual: false
Expected: true
Difference in Case 1, MagneticStarFluxY
The fiducial value is:       -83.294404848633206
The test value is:           -83.294404848633363
The absolute difference is:  1.5631940186722204e-13
The ULP difference is:       11

src/utils/../utils/testing_utilities.h:149: Failure
Value of: areEqual
  Actual: false
Expected: true
Difference in Case 1, MagneticStarFluxZ
The fiducial value is:       -504.8413875424834
The test value is:           -504.84138754248409
The absolute difference is:  6.8212102632969618e-13
The ULP difference is:       12

src/utils/../utils/testing_utilities.h:149: Failure
Value of: areEqual
  Actual: false
Expected: true
Difference in Case 2, MomentumStarFluxZ
The fiducial value is:       -51.343648925162142
The test value is:           -51.3436489251621
The absolute difference is:  4.2632564145606011e-14
The ULP difference is:       6

[  FAILED  ] tMHDHlldInternalStarFluxes.CorrectInputNonDegenerateExpectCorrectOutput (0 ms)

[ RUN      ] tMHDHlldInternalComputeStarState.CorrectInputNonDegenerateExpectCorrectOutput
src/utils/../utils/testing_utilities.h:149: Failure
Value of: areEqual
  Actual: false
Expected: true
Difference in Case 1, EnergyStar
The fiducial value is:       1023.8840191068896
The test value is:           1023.8840191068904
The absolute difference is:  7.9580786405131221e-13
The ULP difference is:       7

[  FAILED  ] tMHDHlldInternalComputeStarState.CorrectInputNonDegenerateExpectCorrectOutput (0 ms)

[  FAILED  ] 3 tests, listed below:
[  FAILED  ] tMHDHlldInternalComputeStarState.CorrectInputNonDegenerateExpectCorrectOutput
[  FAILED  ] tMHDHlldInternalStarFluxes.CorrectInputNonDegenerateExpectCorrectOutput
[  FAILED  ] tMHDHlldInternalDoubleStarFluxes.CorrectInputExpectCorrectOutput

Closed with PR #262


feat(refresher): add pullFactor property

Feature request
I added feature request #15425 and named it pullFactor
If you pull down it's equal to the speed of the user's finger (or cursor). With pullFactor (defaults to 1) the user can set a different value and make the content move faster or slower on pull.
Closes #15425

We discussed this with the team and decided to go with 1 where <1 makes it slower and >1 makes it faster. This keeps it more consistent with others such as progress bar value / buffer. I went ahead and made these changes and added some usage documentation!


Added dockerfile and docker-compose configuration for easy setup

Pull Request Title
Added docker support
Description
Added Dockerfile and docker-compose configuration for easy setup
Type

[ ] Bug Fix
[x] Feature Enhancement
[x] Documentation Update
[ ] Code Refactoring
[ ] Other (please specify):

Proposed Changes

Added build.dockerfile and docker-compose.yml
Updated README.md to include hints to run application using docker

How to Test

Install docker
run docker-compose up
open localhost:80 in your browser

Checklist

[x] The code compiles successfully without any errors or warnings
[x] The changes have been tested and verified
[x] The documentation has been updated (if applicable)
[x] The changes follow the project's coding guidelines and best practices
[x] The commit messages are descriptive and follow the project's guidelines
[ ] This pull request has been linked to the related issue (if applicable)

Compiling Without any issue we can merge this @srbhr

Terminal Output :
[+] Running 1/1
 ! resume-matcher Warning                                                                                4.3s
[+] Building 195.1s (14/14) FINISHED
 => [resume-matcher internal] load build definition from build.dockerfile                                                                                0.0s
 => => transferring dockerfile: 368B                                                                                0.0s
 => [resume-matcher internal] load .dockerignore                                                                                0.0s
 => => transferring context: 2B                                                                                0.0s
 => [resume-matcher internal] load metadata for docker.io/library/python:3.11.0-slim                                                                                5.0s
 => [resume-matcher auth] library/python:pull token for registry-1.docker.io                                                                                0.0s
 => [resume-matcher 1/8] FROM docker.io/library/python:3.11.0-slim@sha256:b5ee5360b357dcd0ab2e1cc5e22fa589b6734bbd55776df6b1700994febf1527                            31.5s
 => => resolve docker.io/library/python:3.11.0-slim@sha256:b5ee5360b357dcd0ab2e1cc5e22fa589b6734bbd55776df6b1700994febf1527                                            0.0s
 => => sha256:23e07e2954939698377b8fe1a859b2d8d0ed4999c7a6da4c983084d50ca4dbe3 1.07MB / 1.07MB                                                                         3.6s
 => => sha256:7a221415bd9bc102fe55706991509a8232d8951333fd2eb5e4546ac7c1293c8c 12.64MB / 12.64MB                                                                       7.7s
 => => sha256:b5ee5360b357dcd0ab2e1cc5e22fa589b6734bbd55776df6b1700994febf1527 1.65kB / 1.65kB                                                                         0.0s
 => => sha256:924c9f87d90abaff0f5cc88c240935ea934323bd24d1ea80bd31caed8bf9cc5e 1.37kB / 1.37kB                                                                         0.0s
 => => sha256:6be1635e79dea843950a77985543461cab28d827d07e058d893b8ec30e6ddf42 7.50kB / 7.50kB                                                                         0.0s
 => => sha256:6064e7e5b6afa4dc711228eddfd250aebac271830dc184c400ce640020bc2cb0 30.06MB / 30.06MB                                                                      29.9s
 => => sha256:bfc052d89fa6dc69a2f95adba9dc5eba6c6dc03fe9fbb0df99e1c540b082cf8c 232B / 232B                                                                             4.4s
 => => sha256:59e7972b564594c664044c01048b262ebaf93b2792f0862a313bf4f25d1fbae9 3.35MB / 3.35MB                                                                        18.9s
 => => extracting sha256:6064e7e5b6afa4dc711228eddfd250aebac271830dc184c400ce640020bc2cb0                                                                              1.0s
 => => extracting sha256:23e07e2954939698377b8fe1a859b2d8d0ed4999c7a6da4c983084d50ca4dbe3                                                                              0.1s
 => => extracting sha256:7a221415bd9bc102fe55706991509a8232d8951333fd2eb5e4546ac7c1293c8c                                                                              0.3s
 => => extracting sha256:bfc052d89fa6dc69a2f95adba9dc5eba6c6dc03fe9fbb0df99e1c540b082cf8c                                                                              0.0s
 => => extracting sha256:59e7972b564594c664044c01048b262ebaf93b2792f0862a313bf4f25d1fbae9                                                                              0.2s
 => [resume-matcher internal] load build context                                                                                5.2s
 => => transferring context: 661.38MB                                                                                5.1s
 => [resume-matcher 2/8] WORKDIR /data                                                                                0.1s
 => [resume-matcher 3/8] COPY . .                                                                                4.1s
 => [resume-matcher 4/8] RUN apt-get update                                                                                6.5s
 => [resume-matcher 5/8] RUN apt-get install -y build-essential python-dev git                                                                                31.3s
 => [resume-matcher 6/8] RUN pip install -U pip setuptools wheel                                                                                3.5s
 => [resume-matcher 7/8] RUN pip install -r requirements.txt                                                                                98.9s
 => [resume-matcher 8/8] RUN python run_first.py                                                                                5.6s
 => [resume-matcher] exporting to image                                                                                8.4s
 => => exporting layers                                                                                8.4s
 => => writing image sha256:6537f0c1b78e285e4fdab608164bf7b708817ddbef0f22a5ed1e0d8ad4b2f3c8                                                                           0.0s
 => => naming to docker.io/library/resume-matcher                                                                                0.0s
[+] Running 2/2
 ✔ Network resume-matcher_default             Created                                                                                0.0s
 ✔ Container resume-matcher-resume-matcher-1  Created                                                                                0.1s
Attaching to resume-matcher-resume-matcher-1
resume-matcher-resume-matcher-1  |
resume-matcher-resume-matcher-1  | Collecting usage statistics. To deactivate, set browser.gatherUsageStats to False.
resume-matcher-resume-matcher-1  |
resume-matcher-resume-matcher-1  |
resume-matcher-resume-matcher-1  |
resume-matcher-resume-matcher-1  |   You can now view your Streamlit app in your browser.
resume-matcher-resume-matcher-1  |
resume-matcher-resume-matcher-1  |   Network URL: http://<IP_ADDRESS>:8501
resume-matcher-resume-matcher-1  |
resume-matcher-resume-matcher-1  |   External URL: http://<IP_ADDRESS>:8501
resume-matcher-resume-matcher-1  |
resume-matcher-resume-matcher-1  |
resume-matcher-resume-matcher-1  | [nltk_data] Downloading package punkt to /root/nltk_data...
resume-matcher-resume-matcher-1  | [nltk_data]   Unzipping tokenizers/punkt.zip.

Perfect @kartik-gupta-ij  and thanks @raditya3  for the PR!!
Thank you both!


There seems to be a bug in benchmark.py.

Issue Description
Hello, I encountered an anomaly while using benchmark.py, where the execution speed during testing was unusually fast. Upon further investigation of benchmark.py, I identified a bug.
In your def benchmark() function, BATCH_SIZE is defaulted to 32, but when calling the benchmark function, BATCH_SIZE is not set to 1.
This results in the dataset's BATCH_SIZE being set to 1, while the model.evaluate(test_dataset, steps=steps_per_epoch) uses steps = len_data // 32 instead of len_data // 1.
Consequently, during testing, only a small amount of test data is read, and due to the absence of shuffle=False in build_dataset, the performance varies with each execution.
Fix
Set BATCH_SIZE to 1 in the def benchmark() function.
Set shuffle=False during the build_dataset step.
Model Weights
I experimented with three sets of model weights:

Pretrained weights provided by you.
Training MetaPolyp from scratch.
Using Pretrained weights and training for 350 epochs.

Among these, option 1 (using your provided Pretrained weights) performed the best.
Test Results Comparison (Kvasir)
Before Fix
dice_coeff: 0.9572
bce_dice_loss: 0.2784
IoU: 0.9183
zero_IoU: 0.9748
mean_squared_error: 0.0184
After Fix
dice_coeff: 0.9049
bce_dice_loss: 0.3448
IoU: 0.8481
zero_IoU: 0.9700
mean_squared_error: 0.0222
Example Usage of benchmark.py
# from save_model.pvt_CAM_channel_att_upscale import build_model
import os
import tensorflow as tf
# from metrics.metrics_last import  iou_metric, MAE, WFbetaMetric, SMeasure, Emeasure,  dice_coef, iou_metric
from metrics.segmentation_metrics import dice_coeff, bce_dice_loss, IoU, zero_IoU, dice_loss
from dataloader.dataloader import build_augmenter, build_dataset, build_decoder
from tensorflow.keras.utils import get_custom_objects
from model import build_model #### from model_research import build_model

os.environ["CUDA_VISIBLE_DEVICES"]="0"

def load_dataset(route):
    X_path = '{}/images/'.format(route)
    Y_path = '{}/masks/'.format(route)
    X_full = sorted(os.listdir(f'{route}/images'))
    Y_full = sorted(os.listdir(f'{route}/masks'))

    X_train = [X_path + x for x in X_full]
    Y_train = [Y_path + x for x in Y_full]

    test_decoder = build_decoder(with_labels=False, target_size=(img_size, img_size), ext='jpg',
                                segment=True, ext2='jpg')
    test_dataset = build_dataset(X_train, Y_train, bsize=BATCH_SIZE, decode_fn=test_decoder,
                                augmentAdv=False, augment=False, augmentAdvSeg=False, shuffle=False, cache=False)
    return test_dataset, len(X_train)

def benchmark(route, model, BATCH_SIZE = 1, save_file_name = "benchmark_result.txt"):
    list_of_datasets = os.listdir(route)
    f = open(save_file_name,"a")
    f.write("\n")
    for datasets in list_of_datasets:
        print(datasets, ":")
        test_dataset, len_data = load_dataset(os.path.join(route,datasets))
        steps_per_epoch = len_data // BATCH_SIZE
        loss, dice_coeff, bce_dice_loss, IoU, zero_IoU, mae = model.evaluate(test_dataset, steps=steps_per_epoch)
        f.write("{}:".format(datasets))
        f.write("dice_coeff: {}, bce_didce_loss: {}, IoU: {}, zero_IoU: {}, mae: {}".format(dice_coeff, bce_dice_loss, IoU, zero_IoU, mae))
        f.write('\n')

if __name__ == "__main__":
    img_size = 256
    BATCH_SIZE = 1
    SEED = 1024
    save_path = "pretrained_model.h5"
    route_data = "./TestDataset/"
    path_to_test_dataset = "./MetaPolyp_Dataset/TestDataset/"
    model = build_model(img_size)
    model.load_weights(save_path)

    model.compile(metrics=[dice_coeff, bce_dice_loss, IoU, zero_IoU, tf.keras.metrics.MeanSquaredError()])

    benchmark(path_to_test_dataset, model)

Hi @Johnsonj0308 , thank you so much for your support to fix the evaluation. I will look for this issue and have an update

Hello, I found in "paper with code" that your dice coefficient currently ranks first, but I also found such problems in benchmark.py. I hope you can correct them in time and give relevant answers, thank you!


[Issue] : Login/signup page

Issue Description
There should be a login/signup page for when we click add to cart or buy now nothing shows..
Are you a Hacktoberfest'21 contributor

[X] Yes, I'am Hacktoberfest'21 contributor

I can create another separate path/file for login/signup then redirect into desired page. Assign me please


Receiving signal but not Transmitting

I have tried this with 3 C1011 boards so far and each time I am able to receive the remote control commands but unable to send them.
I am sure it must be something I am doing wrong but for the life of me can not figure out what it is. I went through the manual and could not see anything I missed so thought I would see if I have missed anything.
I currently have 2 of the 8 pin E07-M1101D boards, one hooked up to a lilygo PoE esp32 and the other connected to a standard Wi-Fi esp32 board. Both connect to the Wi-Fi and allow me to see when a remote is pressed for any of the blinds I have but when I try to pair with a blind nothing happens.
I have tried checking the logs from the opposite esp-Somfy set up and can see no record of the opposite device when I press the prog button on the webpage.
I have also tried following the tuning but this makes no difference on the transmition of the signal. Is it possible I have followed the wiring wrong? I thought it at first but as I was receiving figured I had to have the tx & rx wires the right way.
I also have problems with getting the Ethernet port working on the lilygo but can work on that later, keen to get the connection to the blinds working first.
I have wired it the same as the instructions on the wiki. Is there anywhere I can pull logs from the portal to see if anything is getting mixed up in the back end?

Please double check that pin marked as GDO0 (pin 3 on the E07-M1101D) is connected to the pin you have selected as the TX pin.  I do not have a LilyGO and this information comes from another user.  This very much looks like what you get when the TX pin is not connected to the GDO0 pin of the transceiver.  The transceiver is capable of receiving on either pin.
As an alternative you can try to hook up GPIO4 to GDO0 then set both the RX and TX to GPIO4.  On some ESP32 boards GPIO35 only allows input and LilyGO doesn't seem to list any restrictions on their pins.

Ok, thank you so much, and thank you for coming back so quickly.
I can now see most the commands being sent by Espsomfy, for some reason it doesn’t show the prog command when I click the prog button for the shade, all the other commands I can see.
Is there another way of getting the portal to send the prog button? I’ll upload some screenshots of what the second esp is seeing being transmitted

This is what I see when I use the actual remote connected to the blind.

And this is what I see when I use Espsomfy to transmit codes including clicking the prog button on the web interface to try and program the controller to the blind

As you can see, I see the prog button being pressed on the physical remote but not the Espsomfy remote.

Thanks for troubleshooting this.  I added filters for the up/down/my buttons and didn't add one for the prog command.  This needed to be done to allow the integrated tilt blinds to work.
Please download the SomfyController.ino.esp32.bin and SomfyController.littlefs.bin files from the main repository (not the release) and update your device.  I will be publishing v1.5.1 later today.

Amazing, thanks rstrouse.
That has resolved the issue with the prog signal being sent and I can now pair espsomfy with my blinds.
After powering off for a while and changing some pins I can also confirm the Lilygo Ethernet is now working, it looks like there is an overlap with some pins in the documentation which was causing the issue, when I fixed that and powered the device down for a bit it has all come up and connecting via Ethernet.
Thanks again

Thanks for replying back.  I will be posting a v1.5.1 shortly.  Essentially, this has no impact for you unless you have an 80-bit remote with a thumbwheel but it will include the changes I made earlier.

Thanks for the help.  I am closing this issue so if you have further issues please submit another ticket.

Thanks for fixing so quickly, had the same issue but now working perfectly on our Somfy garage door 😄
love espSomfy - what a great development, thank you.


Main README recommending Warp

The main README is recommending Warp as a high level HTTP server library, but there are other choices and it seems Axum is a better one to recommend because it is developed by Tokio organization similar to Hyper itself.

I happen to think warp is a decent option, but I'm biased :). However, I do agree that if people have no idea what they want, Axum is a better default. So, I'd probably lean towards this adjustment:

Suggest Axum as a great default, especially if you're not sure what you want.
Keep warp as a suggestion, pointing out "it's a little different, meant for people who like Functional Programming and type abuse."
I wish we could point at some sort of directory, such as arewewebyet.org, but it's rather outdated.


Community Meeting October 11th, 2023

Overview
At 10am Eastern, 7am Pacific, 2pm UTC; find your time
Video call link: https://meet.google.com/ved-ivnv-aen
dial-in: (US) +1<PHONE_NUMBER>
PIN: 873 550 579#
More phone numbers: https://tel.meet/ved-ivnv-aen?pin=7872592963542&hs=1
Add topics you'd like to discuss below!

Discussion topic: let's revisit merge criteria for an open community?

Meeting recording: https://drive.google.com/file/d/1-ddw38GUtR5Z0rNwK-XdJpXbl2gNMrye/view?usp=drive_link


Other commands?

I'm a newbie at this. Is there a list of additional commands codes available? I'm doing this for a project kinda need help...Thanks in advance!

Hey @damset2 ! Hope your having a good day! I can help you, just send me an email if you need specific commands<EMAIL_ADDRESS>Anyways, let's get on to the code!
const Discord = require("discord.js");

const client = new Discord.Client();

const config = require("./config.json");

client.on("ready", () => {
    console.log(`Bot has started, with ${client.users.size} users, in ${client.channels.size} channels of ${client.guilds.size} guilds.`);
    client.user.setActivity(`Serving ${client.guilds.size} servers`);
});

client.on("guildCreate", guild => {
    console.log(`New guild joined: ${guild.name} (id: ${guild.id}). This guild has ${guild.memberCount} members!`);
    client.user.setActivity(`Serving ${client.guilds.size} servers`);
});

client.on("guildDelete", guild => {
    console.log(`I have been removed from: ${guild.name} (id: ${guild.id})`);
    client.user.setActivity(`Serving ${client.guilds.size} servers`);
});

client.on("message", async message => {

    if (message.author.bot) return;

    if (message.content.indexOf(config.prefix) !== 0) return;

    const args = message.content.slice(config.prefix.length).trim().split(/ +/g);
    const command = args.shift().toLowerCase();

    if (command === "ping") {
        const m = await message.channel.send("Ping?");
        m.edit(`Pong! Latency is ${m.createdTimestamp - message.createdTimestamp}ms. API Latency is ${Math.round(client.ping)}ms`);
    }

    if (command === "say") {
        const sayMessage = args.join(" ");
        message.delete().catch(O_o => {});
        message.channel.send(sayMessage);
    }

    if (command === "kick") {
        if (!message.member.roles.some(r => ["Administrator", "Moderator"].includes(r.name)))
            return message.reply("Sorry, you don't have permissions to use this!");

        let member = message.mentions.members.first() || message.guild.members.get(args[0]);
        if (!member)
            return message.reply("Please mention a valid member of this server");
        if (!member.kickable)
            return message.reply("I cannot kick this user! Do they have a higher role? Do I have kick permissions?");

        let reason = args.slice(1).join(' ');
        if (!reason) reason = "No reason provided";

        await member.kick(reason)
            .catch(error => message.reply(`Sorry ${message.author} I couldn't kick because of : ${error}`));
        message.reply(`${member.user.tag} has been kicked by ${message.author.tag} because: ${reason}`);

    }

    if (command === "ban") {
        // Most of this command is identical to kick, except that here we'll only let admins do it.
        if (!message.member.roles.some(r => ["Administrator"].includes(r.name)))
            return message.reply("Sorry, you don't have permissions to use this!");

        let member = message.mentions.members.first();
        if (!member)
            return message.reply("Please mention a valid member of this server");
        if (!member.bannable)
            return message.reply("I cannot ban this user! Do they have a higher role? Do I have ban permissions?");

        let reason = args.slice(1).join(' ');
        if (!reason) reason = "No reason provided";

        await member.ban(reason)
            .catch(error => message.reply(`Sorry ${message.author} I couldn't ban because of : ${error}`));
        message.reply(`${member.user.tag} has been banned by ${message.author.tag} because: ${reason}`);
    }

    if (command === "purge") {
        // This command removes all messages from all users in the channel, up to 100.

        // get the delete count, as an actual number.
        const deleteCount = parseInt(args[0], 10);

        // Ooooh nice, combined conditions. <3
        if (!deleteCount || deleteCount < 2 || deleteCount > 100)
            return message.reply("Please provide a number between 2 and 100 for the number of messages to delete");

        // So we get our messages, and delete them. Simple enough, right?
        const fetched = await message.channel.fetchMessages({
            limit: deleteCount
        });
        message.channel.bulkDelete(fetched)
            .catch(error => message.reply(`Couldn't delete messages because of: ${error}`));
    }
});

client.login(config.token);

After this, you will need to do the following:
Setup

Create a new folder somewhere, give it a bot name, aka MyBot or SuperBot
Create a file called app.js, and paste in the contents of app.js (THE STUFF ABOVE).
Create a file called config.json and give it the following content (THE STUFF BELOW), using your real bot token:

{
  "token"  : "MTg-this-IzNzU3OTA5NjA-is.not-DCeFB-a.real-r4DQlO-t0ken-qerT0",
  "prefix" : "+"
}

Thank you!


Control keys are hijacked within Table component

There is a weird bug that Table component actually hijacks keys like left, right, home, end etc. for elements within it.
Example code:
() => {
  const [content, setContent] = useState('Try using control keys');
  return <Table>
    <Box padding={10}>
        <Textarea placeholder="This is a content placeholder" label="Content" name="content" hint="Description line" error={content.length < 5 ? 'Content is too short' : undefined} onChange={e => setContent(e.target.value)}>
      {content}
    </Textarea>
  </Box>
</Table>;
}

Try editing it using arrow keys and home / end keys, they don't work.

Quck workaround:
Add a onKeyDown={e => e.stopPropagation()} to solve this issue. However I hope strapi can fix it

Hello @dzcpy, remove the key handlers is not planned since it is used to allow the keyboard navigation in the table.


Remove References to Staticman Public Instances in Docs

According to https://github.com/eduardoboucas/staticman/issues/317#issuecomment-565250060, the public instance for staticman is no longer supported.
In the docs, there are still references on how to setup staticman using the public instance.
I believe these references should be removed.

You are correct. See #343 for some additional details about the docs. It's on my todo list to fix that but I haven't had time to get to it yet. In the meantime, anyone is welcome to submit a PR on the documentation repo if so inclined.


Consular same-sex marriage service in Romania (for none Romanians)

Trello Card

Content change request affecting all same-sex outcomes except where partner is Romanian. These changes have come about as a result of possibility of holding same sex British Embassy in Romania

Factcheck
Preview Link
GOV.UK
Expected changes

These content changes only affect same sex outcome where partner is not (local) a Romanian

Before

After

Affected Pages:
http://smart-answers-pr-2535.herokuapp.com/marriage-abroad/romania/ceremony_country/partner_british/same_sex
http://smart-answers-pr-2535.herokuapp.com/marriage-abroad/romania/ceremony_country/partner_other/same_sex
http://smart-answers-pr-2535.herokuapp.com/marriage-abroad/romania/third_country/partner_british/same_sex
http://smart-answers-pr-2535.herokuapp.com/marriage-abroad/romania/third_country/partner_other/same_sex
http://smart-answers-pr-2535.herokuapp.com/marriage-abroad/romania/uk/partner_british/same_sex
http://smart-answers-pr-2535.herokuapp.com/marriage-abroad/romania/uk/partner_other/same_sex

Looks good - ready for code review as far as content's concerned :+1:
Here's the original Zendesk ticket.

@sistephens thanks for the heads up

👍  - Apart from the two comments


Initialize logger if no log_dir is configured during log purge

The logger can end up not being _configured if this is not called. This can occure when log level is set to a stricter level like error and no errors occur, then when we call the purge_logs function our handler could be un-configured and not have a log_dir attribute -- resulting in the function bailing out.

Maintainer merge checklist

[ ] Title is descriptive/clear for inclusion in release notes.
[ ] Applied a Component: xxx label.
[ ] Applied the api-deprecation or api-break label.
[ ] Applied the release-highlight label to be highlighted in release notes.
[ ] Added to the milestone version it was merged into.
[ ] Unittests are included in PR.
[ ] Properly documented, including versionadded, versionchanged as needed.

Hi @craigkai !
Sorry for the late reply, but we were really busy with the new release process.
How do you feel about adding a test that exercises this issue so we can ensure that does not happen again in the future?

I'll see what I can do!

Ping @craigkai !


Extract sqlite.js from Tofino

This will allow us to use the promisified sqlite interface from Datomish.

Coverage decreased (-0.4%) to 42.407% when pulling 0afc02037ad9ee8df75cc4ac181cabbb58e4e5ba on rnewman:rnewman/extract-sqlite into b7fd1012e06c742d5f1b044e3f1f848a10bd86ff on mozilla:master.

In general, I'm fine with this.
Right now, my thinking is that Clojure(Script)'s core.async is interesting and strictly more powerful than promises, and therefore worth investigating for our internal usage.  There are channels with promise semantics (promise-chan) and there are promise-to-channel wrappers (https://github.com/jamesmacaulay/cljs-promises).  I'm not fluent with channels quite yet but they me be better than committing to promises immediately.
That discussion impacts this patch to say: I'm not sure we want a promise-sqlite; we might want a channel-sqlite.

@ncalexan: indeed. Either way that means breaking this stuff out — firstly so that Datomish can start using it as well as Tofino (e.g., for Places import), and secondly so that Datomish can stop using it later if we choose.

@rnewman ah, I'd forgotten about Tofino's non-UAS SQLite desires.  Right-o.

Coverage decreased (-0.4%) to 42.407% when pulling badd3ad70a4ca29049c6bf4c5db2a90223642242 on rnewman:rnewman/extract-sqlite into b7fd1012e06c742d5f1b044e3f1f848a10bd86ff on mozilla:master.

64e33ebc62bf14c5f57993c9a5f70528eceb717e


Fixed sizing style definition and types

Summary
I appreciate the recent addition of shorthands for sizing props: maxH, maxW, minH and minW. However, there's a small thing missing to make it work correctly :)
The final CSS output has incorrect property, because the cssProperty is not set explicitly in the style function's config object. Also, the types were copy-pasted incorrectly. Fixed those too for consistency.
Test plan
These props

Turn into these styles

Thanks!


Restore does not work with TTL = 0

Problem
Per the RESTORE documentation

If ttl is 0 the key is created without any expire, otherwise the specified expire time (in milliseconds) is set.

Documentation here.
https://redis.io/commands/restore
However, if you try to use the restore function with a zero TTL, you get the following error
redis.exceptions.ResponseError: Invalid TTL value, must be >= 0
While this is a valid check elsewhere, this is not a valid condition for the RESTORE command.

My bad, the problem is in fact passing -1 to restore.
I got the value from TTL, which reports -1 when the TTL is not set. So you need to convert -1 into 0 for restore to succeed.


Load generator returns connection error in GKE cluster

Bug Report
Which version of the demo you are using? (please provide either a specific
32add44
Symptom

Load generator running headless returns connection refused when it runs in GKE cluster

What is the expected behavior?
What do you expect to see?

Load generator running headless should generate the traces for r number of users for given period of time.

What did you see instead?

I get connection refused when it runs in GKE cluster.

Reproduce
Reproduction for load generator running headless in GKE cluster

Start a collector in GKE cluster which is receiving request on 4718. The collector endpoint is otel-collector.platform-opentelemetry.svc.cluster.local:4318/v1/traces
Created a docker image from the Dockerfile and running a pod in the same namespace with below settings.

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    run: locust
  name: locust
  namespace: platform-opentelemetry
spec:
  automountServiceAccountToken: true
  serviceAccount: otel-collector-sa
  containers:
  - image: xyz-docker.artifactory.gcp.xyz/observability/otel/loadgenerator:0.0.2
    command: ["locust"]
    ports:
    - containerPort: 8089
    name: locust
    resources: {}
    env:
    - name: LOCUST_WEB_PORT
      value: "8089"
    - name: LOCUST_USERS
      value: "10"
    - name: LOCUST_HOST
      value: "frontend:8080"
    - name: LOCUST_WEB_HOST
      value: loadgenerator
    - name: LOCUST_HEADLESS
      value: "true"
    - name: LOCUST_AUTOSTART
      value: "true"
    - name: OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
      value: "http://otel-collector.platform-opentelemetry.svc.cluster.local:4318/v1/traces"
    - name: OTEL_RESOURCE_ATTRIBUTES
      value: "service.namespace=platform-opentelemetry"
    - name: OTEL_SERVICE_NAME
      value: "otel-loadgenerator"
    - name: PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION
      value: "python"
  dnsPolicy: ClusterFirst
  restartPolicy: Always
status: {}

No frontend service or frontend service proxy has been started. I am unsure of what the environment variables LOCUST_HOST, LOCUST_WEB_PORT will do.
I can curl from the pod to the service end point otel-collector.platform-opentelemetry.svc.cluster.local:4318/v1/traces but when load generator starts, this throw following error.

sdk54@C02FVFZ2MD6M src % k logs -f locust
[2023-02-26 20:10:03,255] locust/WARNING/locust.main: The --autostart argument is implied by --headless, no need to set both.
[2023-02-26 20:10:03,255] locust/INFO/locust.main: No run time limit set, use CTRL+C to interrupt
[2023-02-26 20:10:03,256] locust/INFO/locust.main: Starting Locust 2.14.0
Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00

[2023-02-26 20:10:03,257] locust/INFO/locust.runners: Ramping to 10 users at a rate of 1.00 per second
Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run
  File "/usr/local/lib/python3.11/site-packages/locust/user/users.py", line 176, in run_user
    user.run()
  File "/usr/local/lib/python3.11/site-packages/locust/user/users.py", line 142, in run
    self.on_start()
  File "/usr/src/app/locustfile.py", line 130, in on_start
    self.index()
  File "/usr/src/app/locustfile.py", line 69, in index
    self.client.get("/")
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/locust/clients.py", line 135, in request
    response = self._send_request_safe_mode(method, url, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/locust/clients.py", line 203, in _send_request_safe_mode
    return super().request(method, url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 128, in instrumented_request
    return _instrumented_requests_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 245, in _instrumented_requests_call
    raise exception.with_traceback(exception.__traceback__)
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 209, in _instrumented_requests_call
    result = call_wrapped()  # *** PROCEED
             ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 126, in call_wrapped
    return wrapped_request(self, method, url, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 148, in instrumented_send
    return _instrumented_requests_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 159, in _instrumented_requests_call
    return call_wrapped()
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 146, in call_wrapped
    return wrapped_send(self, request, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 695, in send
    adapter = self.get_adapter(url=request.url)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 792, in get_adapter
    raise InvalidSchema(f"No connection adapters were found for {url!r}")
requests.exceptions.InvalidSchema: No connection adapters were found for 'frontend:8080/'
2023-02-26T20:10:03Z <Greenlet at 0x7f5fbbb38ea0: run_user(<locustfile.WebsiteUser object at 0x7f5fbbb35b90>)> failed with InvalidSchema

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run
  File "/usr/local/lib/python3.11/site-packages/locust/user/users.py", line 176, in run_user
    user.run()
  File "/usr/local/lib/python3.11/site-packages/locust/user/users.py", line 142, in run
    self.on_start()
  File "/usr/src/app/locustfile.py", line 130, in on_start
    self.index()
  File "/usr/src/app/locustfile.py", line 69, in index
    self.client.get("/")
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/locust/clients.py", line 135, in request
    response = self._send_request_safe_mode(method, url, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/locust/clients.py", line 203, in _send_request_safe_mode
    return super().request(method, url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 128, in instrumented_request
    return _instrumented_requests_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 245, in _instrumented_requests_call
    raise exception.with_traceback(exception.__traceback__)
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 209, in _instrumented_requests_call
    result = call_wrapped()  # *** PROCEED
             ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 126, in call_wrapped
    return wrapped_request(self, method, url, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 148, in instrumented_send
    return _instrumented_requests_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 159, in _instrumented_requests_call
    return call_wrapped()
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 146, in call_wrapped
    return wrapped_send(self, request, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 695, in send
    adapter = self.get_adapter(url=request.url)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 792, in get_adapter
    raise InvalidSchema(f"No connection adapters were found for {url!r}")
requests.exceptions.InvalidSchema: No connection adapters were found for 'frontend:8080/'
2023-02-26T20:10:04Z <Greenlet at 0x7f5fbbb3b100: run_user(<locustfile.WebsiteUser object at 0x7f5fbbb36750>)> failed with InvalidSchema

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run
  File "/usr/local/lib/python3.11/site-packages/locust/user/users.py", line 176, in run_user
    user.run()
  File "/usr/local/lib/python3.11/site-packages/locust/user/users.py", line 142, in run
    self.on_start()
  File "/usr/src/app/locustfile.py", line 130, in on_start
    self.index()
  File "/usr/src/app/locustfile.py", line 69, in index
    self.client.get("/")
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 600, in get
    return self.request("GET", url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/locust/clients.py", line 135, in request
    response = self._send_request_safe_mode(method, url, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/locust/clients.py", line 203, in _send_request_safe_mode
    return super().request(method, url, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 128, in instrumented_request
    return _instrumented_requests_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 245, in _instrumented_requests_call
    raise exception.with_traceback(exception.__traceback__)
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 209, in _instrumented_requests_call
    result = call_wrapped()  # *** PROCEED
             ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 126, in call_wrapped
    return wrapped_request(self, method, url, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 148, in instrumented_send
    return _instrumented_requests_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 159, in _instrumented_requests_call
    return call_wrapped()
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 146, in call_wrapped
    return wrapped_send(self, request, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 695, in send
    adapter = self.get_adapter(url=request.url)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 792, in get_adapter
    raise InvalidSchema(f"No connection adapters were found for {url!r}")
requests.exceptions.InvalidSchema: No connection adapters were found for 'frontend:8080/'
2023-02-26T20:10:04Z <Greenlet at 0x7f5fbbb3b240: run_user(<locustfile.WebsiteUser object at 0x7f5fbbb84c50>)> failed with InvalidSchema

Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00

[2023-02-26 20:10:13,292] locust/ERROR/opentelemetry.sdk.trace.export: Exception while exporting Span batch.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 95, in create_connection
    raise err
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/connection.py", line 85, in create_connection
    sock.connect(sa)
  File "/usr/local/lib/python3.11/site-packages/gevent/_socketcommon.py", line 607, in connect
    raise _SocketError(err, strerror(err))
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
                       ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 398, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 239, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/usr/local/lib/python3.11/http/client.py", line 1282, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.11/http/client.py", line 1328, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.11/http/client.py", line 1277, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.11/http/client.py", line 1037, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.11/http/client.py", line 975, in send
    self.connect()
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 205, in connect
    conn = self._new_conn()
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f5fbbbbf4d0>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/sdk/trace/export/__init__.py", line 373, in _export_batch
    self.span_exporter.export(self.spans_list[:idx])  # type: ignore
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 153, in export
    resp = self._export(serialized_data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py", line 124, in _export
    return self._session.post(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 635, in post
    return self.request("POST", url, data=data, json=json, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 128, in instrumented_request
    return _instrumented_requests_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 159, in _instrumented_requests_call
    return call_wrapped()
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 126, in call_wrapped
    return wrapped_request(self, method, url, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 587, in request
    resp = self.send(prep, **send_kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 148, in instrumented_send
    return _instrumented_requests_call(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 159, in _instrumented_requests_call
    return call_wrapped()
           ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/opentelemetry/instrumentation/requests/__init__.py", line 146, in call_wrapped
    return wrapped_send(self, request, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 701, in send
    r = adapter.send(request, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 547, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

Type     Name                                                                          # reqs      # fails |    Avg     Min     Max    Med |   req/s  failures/s
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
--------|----------------------------------------------------------------------------|-------|-------------|-------|-------|-------|-------|--------|-----------
         Aggregated                                                                         0     0(0.00%) |      0       0       0      0 |    0.00        0.00

We will close this issue if:

The steps you provided are complex.
If we can not reproduce the behavior you're reporting.

Additional Context
Add any other context about the problem here.

@sudhirkumar05 are you able to reproduce this problem locally (e.g. using docker-compose)?

I believe you are seeing these errors because you don't have a scheme prefix for the LOCUST_HOST environment variable. It should be http://frontend:8080

prefixing http:// helped me to resolve the issue, i can generate load using http collector, however failing for grpc as of now


Support UniqueConstraint for sqldiff

It seems that manage sqldiff does not recognize unique_together defined using constraints.
This works
    class Meta:
        unique_together = ["product", "tag"]

This does not work:
    class Meta:
        constraints = [
            UniqueConstraint(fields=["product", "tag"], name="unique_product_tag")
        ]

The documentation says unique_together may be deprecated in the future,
https://docs.djangoproject.com/en/3.0/ref/models/options/#unique-together
it may be worth it to support UniqueConstraint.

Confirming with version 3.1.5 and Django 4.0.3

Also CheckConstraint is treated as index and is proposed for removal.
Constraint description:
            models.CheckConstraint(check=models.Q(datn__lte=models.F("datk")), name="st_address_ch_dat"),

Proposed SQL:
DROP INDEX "st_adres_ch_dat";

How it looks in DBeaver:

Also field_name is not respected on unique_together
Model definition:
class StChemDog(models.Model):
    ch_kod = models.ForeignKey(Chemie, models.DO_NOTHING, db_column="ch_kod")
    kod_dog = models.ForeignKey(Dogovor, models.DO_NOTHING, db_column="kod_dog")

    class Meta:
        db_table = "st_chem_dog"
        unique_together = (("kod_dog", "ch_kod"),)

How it looks in DBeaver:

Proposed SQL:
ALTER TABLE "st_chem_dog"
	DROP CONSTRAINT "st_chem_dog_un_dog_ch";
ALTER TABLE "st_chem_dog"
	ADD CONSTRAINT "st_chem_dog_kod_dog_id_ch_kod_id_dab79e14_uniq" UNIQUE ("kod_dog_id", "ch_kod_id");

Dropping existing constraint is OK I guess as name does not match, but trying to change field names to kod_dog_id and ch_kod_id is definitely not right.


Fix test cases

All the test cases are fixed, and all function test cases have achieved 100% coverage on jest, except the following function:

onDisplay.js
icalToJson
This is because this function relies on external JSON data and HTML elements, which makes it hard to control all scenarios for testing coverage.

Fix issue #19

Thank you Vishva for informing me that there is a styling issue in my code, I did not realise that my prettier setting is changed. I have fixed the styling issue. I always linked my pull request to a issue.


Use the same noescape that the Go standard library in ioutilx

Fix a typo of preferred in the Read and Write helpers.
Use the same noescape implementation used by standard library code, instead of a similar derivation. See https://golang.org/src/strings/builder.go#L20

Follow this checklist to help us incorporate your contribution quickly and easily:

[ ] Format the pull request title like [BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replace BEAM-XXX with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
[ ] If this contribution is large, please file an Apache Individual Contributor License Agreement.

It will help us expedite review of your Pull Request if you tag someone (e.g. @username) to look at it.
 Thank you.


How to set text alignment?

I need to left align text in picker. How it can be made?

Does picker.setTextAlignment(TEXT_ALIGNMENT_VIEW_START) fit your needs ?

Emm well did you implement it? Because I can't find it anywhere in your documents / source and I need alignment!
Thanks


chore(Algebra/Order/Group/Defs): 3 line breaks

Move 3 line breaks, so that they no longer separate inputs to Co*variantClasses.

@eric-wieser, given bors rejection, can I sneak in one more line break fix?

bors d+