Demo : Unexpected result in some queries

I have two  problems   with the master branch(commit: ae3c02fb3)
my steps:

use HDFSParquetImporter to import from hive to hudi

use  HoodieDeltaStreamer to import new data from kafka.(I add a option to allow missing checkpointStr)
the config is same as #779, with --disable-compaction.
And then
 select distinct _hoodie_commit_time from rt_table/ro_table  only return the first the commit time (use max() to ensure no newer commits return);  But there are newer .deltacommit file in the .hoodie folder.

restart the spark job. open the spark UI, will find that   the job hangs at collect at HoodieMergeOnReadTable.java:318 (It hangs every time)

org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45)
com.uber.hoodie.table.HoodieMergeOnReadTable.rollback(HoodieMergeOnReadTable.java:318)
com.uber.hoodie.HoodieWriteClient.doRollbackAndGetStats(HoodieWriteClient.java:884)
com.uber.hoodie.HoodieWriteClient.rollbackInternal(HoodieWriteClient.java:962)
com.uber.hoodie.HoodieWriteClient.rollback(HoodieWriteClient.java:773)
com.uber.hoodie.HoodieWriteClient.rollbackInflightCommits(HoodieWriteClient.java:1182)
com.uber.hoodie.HoodieWriteClient.startCommitWithTime(HoodieWriteClient.java:1050)
com.uber.hoodie.HoodieWriteClient.startCommit(HoodieWriteClient.java:1043)
com.uber.hoodie.utilities.deltastreamer.DeltaSync.startCommit(DeltaSync.java:406)
com.uber.hoodie.utilities.deltastreamer.DeltaSync.writeToSink(DeltaSync.java:332)
com.uber.hoodie.utilities.deltastreamer.DeltaSync.syncOnce(DeltaSync.java:227)
com.uber.hoodie.utilities.deltastreamer.HoodieDeltaStreamer$DeltaSyncService.lambda$startService$0(HoodieDeltaStreamer.java:382)
java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)

@eisig :
Regarding (2), Can you list the timeline (.hoodie folder) and all partition path folders ? Also, for the latest committed deltaCommit, can you cat the file and attach.

@eisig : Also regarding (3) How long did you notice it to be in hanging state ? My guess is the rollback is progressing but slowly (https://issues.apache.org/jira/browse/HUDI-169). Can you check executor logs corresponding to active tasks and see if this is the case ?

@bvaradar
Here the data
https://raw.githubusercontent.com/eisig/sample-files/master/hudi-789/metadata.txt
https://raw.githubusercontent.com/eisig/sample-files/master/hudi-789/data-list.txt

@bvaradar It is a file not found exception cause (3).
Maybe you PR https://github.com/apache/incubator-hudi/pull/788 will fix this.

@bvaradar  About the timeline. For MOR table.

The rt view should alway return the lastest deltacommit data. (this issue)
The ro view should only return the lastest compact commit data.

Am I not understanding correctly?
The ro view would return the lastest commit data without compact.In  #779

@eisig yeah your understanding is correct. Regarding RT query, can you enable debug in your spark shell and give the driver logs.  It would also help if you run the same query against 0.4.7 . Will be online in an hour to debug further

@eisig The PR https://github.com/apache/incubator-hudi/pull/788 should fix the rollback issue.
For the RT query, are you ingesting data into newer partitions ? If yes, those partitions will need to be registered with the hive metastore using the hive sync tool.

@bvaradar I make (3) pass, when new commit is add to the metadata,  it seems  to be correct eventually. I need to reproduce this problem again.
@n3nash The  data is partitioned by year, so  all the partitions will be created in steps(1)

In (2), it seems like you are not seeing the delta commit data reflected? again how do we reconcile this with duplicates you were reporting on #779 ? do you think they are related?

@eisig : It would be great if we could set an 1-1 session to understand the issue and debug this and the other ones. Can you  ping me in slack (vbalaji) ?

@bvaradar  please add me to to slack channel, my email<EMAIL_ADDRESS>thx

The timeline seems not work as expect .
I rerun the demo https://hudi.apache.org/docker_demo.html
Step 6(a) the ro and rt view is same.

@eisig : Added you to slack. Ping me (vbalaji) when you are online

The timeline seems not work as expect .
I rerun the demo https://hudi.apache.org/docker_demo.html
Step 6(a) the mor table ro and rt view return the same result.

@eisig : I have seen this happen when we miss passing the flag "--disable-compaction" as part of deltastreamer run. When you run the demo (step - 2), after running deltastreamer for MOR, verify there is only one ".deltacommit" file and no ".commit" in http://namenode:50070/explorer.html#/user/hive/warehouse/stock_ticks_mor/.hoodie
The demo link you pointed was recently updated to reflect it.

The result of the demo is very strange
(Step 6(a))
 select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_mor_rt where  symbol = 'GOOG';
 select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_mor where  symbol = 'GOOG';

return as demo
BUT!
select a.key,a.ts, b.ts from stock_ticks_mor a join stock_ticks_mor_rt b  on a.key=b.key where a.ts != b.ts
...
+--------+-------+-------+--+
| a.key  | a.ts  | b.ts  |
+--------+-------+-------+--+
+--------+-------+-------+--+

0: jdbc:hive2://hiveserver:10000> select a.key,a.ts,b.ts from stock_ticks_mor_rt a join stock_ticks_mor b on a.key = b.key where a.key= 'GOOG_2018-08-31 10';
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.8.4/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Execution log at: /tmp/root/root_20190718091316_ec40e8f2-be17-4450-bb75-8db9f4390041.log
2019-07-18 09:13:20 Starting to launch local task to process map join;  maximum memory = 477626368
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2019-07-18 09:13:21 Dump the side-table for tag: 0 with group count: 1 into file: file:/tmp/root/60ae1624-3514-4ddd-9bc1-5d2349d922d6/hive_2019-07-18_09-13-16_658_8306103829282410332-1/-local-10005/HashTable-Stage-3/MapJoin-mapfile50--.hashtable
2019-07-18 09:13:21 Uploaded 1 File to: file:/tmp/root/60ae1624-3514-4ddd-9bc1-5d2349d922d6/hive_2019-07-18_09-13-16_658_8306103829282410332-1/-local-10005/HashTable-Stage-3/MapJoin-mapfile50--.hashtable (317 bytes)
2019-07-18 09:13:21 End of local task; Time Taken: 1.688 sec.
+---------------------+----------------------+----------------------+--+
|        a.key        |         a.ts         |         b.ts         |
+---------------------+----------------------+----------------------+--+
| GOOG_2018-08-31 10  | 2018-08-31 10:29:00  | 2018-08-31 10:29:00  |
+---------------------+----------------------+----------------------+--+
1 row selected (7.207 seconds)
0: jdbc:hive2://hiveserver:10000> select a.key,a.ts,b.ts from stock_ticks_mor a join stock_ticks_mor_rt b on a.key = b.key where a.key= 'GOOG_2018-08-31 10';
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/hive/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/hadoop-2.8.4/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Execution log at: /tmp/root/root_20190718091348_72a5fc30-fc04-41c1-b2e3-5f943e4d5c08.log
2019-07-18 09:13:51 Starting to launch local task to process map join;  maximum memory = 477626368
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2019-07-18 09:13:53 Dump the side-table for tag: 0 with group count: 1 into file: file:/tmp/root/60ae1624-3514-4ddd-9bc1-5d2349d922d6/hive_2019-07-18_09-13-48_027_3613368446029280476-1/-local-10005/HashTable-Stage-3/MapJoin-mapfile60--.hashtable
2019-07-18 09:13:53 Uploaded 1 File to: file:/tmp/root/60ae1624-3514-4ddd-9bc1-5d2349d922d6/hive_2019-07-18_09-13-48_027_3613368446029280476-1/-local-10005/HashTable-Stage-3/MapJoin-mapfile60--.hashtable (317 bytes)
2019-07-18 09:13:53 End of local task; Time Taken: 2.36 sec.
+---------------------+----------------------+----------------------+--+
|        a.key        |         a.ts         |         b.ts         |
+---------------------+----------------------+----------------------+--+
| GOOG_2018-08-31 10  | 2018-08-31 10:59:00  | 2018-08-31 10:59:00  |
+---------------------+----------------------+----------------------+--+

Stop the demo (./stop_demo.sh)
Add a new line to batch_2.json
{"volume": 9205, "symbol": "TEST", "ts": "2018-08-31 10:59:00", "month": "08", "high": 153.48, "low": 153.36, "key": "TEST_2018-08-31 10", "year": 2018, "date": "2018/08/31", "close": 153.48, "open": 153.36, "day": "31"}

rerun the demo, and the rt and ro view is the same!
0: jdbc:hive2://hiveserver:10000>  select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_mor_rt where  symbol = 'GOOG';
+----------------------+---------+----------------------+---------+------------+-----------+--+
| _hoodie_commit_time  | symbol  |          ts          | volume  |    open    |   close   |
+----------------------+---------+----------------------+---------+------------+-----------+--+
|<PHONE_NUMBER>2114       | GOOG    | 2018-08-31 09:59:00  | 6330    | 1230.5     | 1230.02   |
|<PHONE_NUMBER>2214       | GOOG    | 2018-08-31 10:59:00  | 9021    | 1227.1993  | 1227.215  |
+----------------------+---------+----------------------+---------+------------+-----------+--+
2 rows selected (0.487 seconds)
0: jdbc:hive2://hiveserver:10000>  select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_mor where  symbol = 'GOOG';
+----------------------+---------+----------------------+---------+------------+-----------+--+
| _hoodie_commit_time  | symbol  |          ts          | volume  |    open    |   close   |
+----------------------+---------+----------------------+---------+------------+-----------+--+
|<PHONE_NUMBER>2114       | GOOG    | 2018-08-31 09:59:00  | 6330    | 1230.5     | 1230.02   |
|<PHONE_NUMBER>2214       | GOOG    | 2018-08-31 10:59:00  | 9021    | 1227.1993  | 1227.215  |
+----------------------+---------+----------------------+---------+------------+-----------+--+

the demo is clean build ( mvn clean install -DskipTests) with the lastest master branch( 6efa16317c0f0) without any change.

It may also be:
(But I can not reproduce this)
select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_mor_rt limit 10;
+----------------------+---------+----------------------+---------+-------+--------+--+
| _hoodie_commit_time  | symbol  |          ts          | volume  | open  | close  |
+----------------------+---------+----------------------+---------+-------+--------+--+
|<PHONE_NUMBER>1622       | NULL    | 2018-08-31 10:58:00  | NULL    | NULL  | NULL   |
|<PHONE_NUMBER>1407       | NULL    | 2018-08-31 09:58:00  | NULL    | NULL  | NULL   |
|<PHONE_NUMBER>1622       | NULL    | 2018-08-31 10:58:00  | NULL    | NULL  | NULL   |
|<PHONE_NUMBER>1407       | NULL    | 2018-08-31 09:59:00  | NULL    | NULL  | NULL   |
|<PHONE_NUMBER>1407       | NULL    | 2018-08-31 09:51:00  | NULL    | NULL  | NULL   |
|<PHONE_NUMBER>1407       | NULL    | 2018-08-31 09:59:00  | NULL    | NULL  | NULL   |
|<PHONE_NUMBER>1407       | NULL    | 2018-08-31 09:54:00  | NULL    | NULL  | NULL   |
|<PHONE_NUMBER>1407       | NULL    | 2018-08-31 09:59:00  | NULL    | NULL  | NULL   |
|<PHONE_NUMBER>1622       | NULL    | 2018-08-31 10:59:00  | NULL    | NULL  | NULL   |
|<PHONE_NUMBER>1407       | NULL    | 2018-08-31 09:53:00  | NULL    | NULL  | NULL   |
+----------------------+---------+----------------------+---------+-------+--------+--+

0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close  from stock_ticks_mor limit 10;
+----------------------+---------+----------------------+---------+---------+---------+--+
| _hoodie_commit_time  | symbol  |          ts          | volume  |  open   |  close  |
+----------------------+---------+----------------------+---------+---------+---------+--+
|<PHONE_NUMBER>1407       | NIHD    | 2018-08-31 10:29:00  | 2125    | 5.67    | 5.67    |
|<PHONE_NUMBER>1407       | CDMO    | 2018-08-31 09:58:00  | 8577    | 7.02    | 7.07    |
|<PHONE_NUMBER>1407       | MRTX    | 2018-08-31 10:28:00  | 1844    | 56.15   | 56.15   |
|<PHONE_NUMBER>1407       | NTRA    | 2018-08-31 09:59:00  | 1326    | 27.45   | 27.45   |
|<PHONE_NUMBER>1407       | DRNA    | 2018-08-31 09:51:00  | 334     | 16.0    | 16.0    |
|<PHONE_NUMBER>1407       | STAA    | 2018-08-31 09:59:00  | 1709    | 47.95   | 47.95   |
|<PHONE_NUMBER>1407       | LGCY    | 2018-08-31 09:54:00  | 2830    | 5.37    | 5.37    |
|<PHONE_NUMBER>1407       | MDB     | 2018-08-31 09:59:00  | 880     | 71.349  | 71.349  |
|<PHONE_NUMBER>1407       | TRHC    | 2018-08-31 10:29:00  | 110     | 86.95   | 86.95   |
|<PHONE_NUMBER>1407       | BAND    | 2018-08-31 09:53:00  | 305     | 45.08   | 45.08   |
+----------------------+---------+----------------------+---------+---------+---------+--+

@bvaradar I  found out  why (2)
select distinct _hoodie_commit_time from rt_table/ro_table only return the first the commit time (use max() to ensure no newer commits return); But there are newer .deltacommit file in the .hoodie folder.

happened.
In steps 1
use HDFSParquetImporter to import from hive to hudi, The data is newer than that in kafka.
Sorry, it's my mistake.

@bhasudha : @eisig is seeing some issues in master branch with the docker setup.
Relevant Info:
https://github.com/apache/incubator-hudi/issues/789#issuecomment-512740619
https://github.com/apache/incubator-hudi/issues/789#issuecomment-512741943

@eisig I am trying to reproduce the issue. Will update soon.

@bhasudha  any update about this?

@eisig I have been trying to reproduce this for sometime. It's very intermittent in my setup. Are you able to deterministically reproduce this ?

@bhasudha It happens every time.

https://github.com/apache/incubator-hudi/issues/789#issuecomment-512740619
https://github.com/apache/incubator-hudi/issues/789#issuecomment-512741943

(2) is import to me. If it's  by design, how to use the MERGE_ON_READ ro view?

Weird that it is intermittent.  @bhasudha lets meet and take a stab at this sometime.. this also blocks #751  and related efforts , which blocks spark upgrade which blocks timestamp support :)

I can reproduce this actually and what I see is that the user columns are removed off the column projection list,before being passed on to AbstractRealtimeRecordReader ?
2019-07-31T10:37:19,108 INFO  [d71b59a8-15fd-4339-a60b-158fa67a3901 HiveServer2-Handler-Pool: Thread-45]: realtime.HoodieRealtimeInputFormat (HoodieRealtimeInputFormat.java:getRecordReader(223)) - Before adding Hoodie columns, Projections :_hoodie_commit_time,volume,ts,symbol,close,open, Ids :0,5,6,7,14,15
2019-07-31T10:37:19,108 INFO  [d71b59a8-15fd-4339-a60b-158fa67a3901 HiveServer2-Handler-Pool: Thread-45]: realtime.HoodieRealtimeInputFormat (HoodieRealtimeInputFormat.java:getRecordReader(235)) - Creating record reader with readCols :_hoodie_commit_time,volume,ts,symbol,close,open,_hoodie_record_key,_hoodie_partition_path, Ids :0,5,6,7,14,15,2,3
2019-07-31T10:37:19,200 INFO  [d71b59a8-15fd-4339-a60b-158fa67a3901 HiveServer2-Handler-Pool: Thread-45]: realtime.AbstractRealtimeRecordReader (AbstractRealtimeRecordReader.java:<init>(97)) - cfg ==> ts,symbol,_hoodie_record_key,_hoodie_commit_time,_hoodie_partition_path
2019-07-31T10:37:19,200 INFO  [d71b59a8-15fd-4339-a60b-158fa67a3901 HiveServer2-Handler-Pool: Thread-45]: realtime.AbstractRealtimeRecordReader (AbstractRealtimeRecordReader.java:<init>(98)) - columnIds ==> 6,7,2,0,3
2019-07-31T10:37:19,200 INFO  [d71b59a8-15fd-4339-a60b-158fa67a3901 HiveServer2-Handler-Pool: Thread-45]: realtime.AbstractRealtimeRecordReader (AbstractRealtimeRecordReader.java:<init>(99)) - partitioningColumns ==> dt

Seems like a regression to me.. is HUDI-151 related?

With disabling of these static variables introduced, I can get the query to work now..
diff --git a/hoodie-hadoop-mr/src/main/java/com/uber/hoodie/hadoop/realtime/HoodieRealtimeInputFormat.java b/hoodie-hadoop-mr/src/main/java/com/uber/hoodie/hadoop/realtime/HoodieRealtimeInputFormat.java
index 14263738..00c36e26 100644
--- a/hoodie-hadoop-mr/src/main/java/com/uber/hoodie/hadoop/realtime/HoodieRealtimeInputFormat.java
+++ b/hoodie-hadoop-mr/src/main/java/com/uber/hoodie/hadoop/realtime/HoodieRealtimeInputFormat.java
@@ -208,11 +208,13 @@ public class HoodieRealtimeInputFormat extends HoodieInputFormat implements Conf
         HOODIE_COMMIT_TIME_COL_POS);
     configuration = addProjectionField(configuration, HoodieRecord.PARTITION_PATH_METADATA_FIELD,
         HOODIE_PARTITION_PATH_COL_POS);
+    /*
     if (!isReadColumnsSet) {
       READ_COLUMN_IDS = configuration.get(ColumnProjectionUtils.READ_COLUMN_IDS_CONF_STR);
       READ_COLUMN_NAMES = configuration.get(ColumnProjectionUtils.READ_COLUMN_NAMES_CONF_STR);
       isReadColumnsSet = true;
     }
+    */
     return configuration;
   }

@@ -241,8 +243,8 @@ public class HoodieRealtimeInputFormat extends HoodieInputFormat implements Conf
             + split);

     // Reset the original column ids and names
-    job.set(ColumnProjectionUtils.READ_COLUMN_IDS_CONF_STR, READ_COLUMN_IDS);
-    job.set(ColumnProjectionUtils.READ_COLUMN_NAMES_CONF_STR, READ_COLUMN_NAMES);
+    /*job.set(ColumnProjectionUtils.READ_COLUMN_IDS_CONF_STR, READ_COLUMN_IDS);
+    job.set(ColumnProjectionUtils.READ_COLUMN_NAMES_CONF_STR, READ_COLUMN_NAMES);*/

     return new HoodieRealtimeRecordReader((HoodieRealtimeFileSplit) split, job,
         super.getRecordReader(split, job, reporter));

those static variables seem to be fixing for some hive version, while breaking hive 2.x.. I dont recall the reasoning behind it. @n3nash ? But, high level caching these values across queries in a static variables does not make a ton of sense to me.. @bvaradar  as well for input

Oh this could also be causing the join issue then?

@n3nash is debugging the join issue, which seems different?

I got this issue when I use same adhoc 1 or 2 terminal to run the steps instead of exiting the current terminal and use one after each main steps per instruction.
May be some environmental variables override issue but not sure.
Please try to execute all the steps in single adhoc 1 or 2 terminal without exiting to reproduce the issue.

Should be fixed now.


Symfony 3.4 support

Description, reason for the PR: upgrade of the main repository to Symfony 3.4
New feature: No
BC breaks: No
Fixes issues: None
Standards and tests pass: Yes
License: MIT

Hi, seems okay, you can merge :+1:


Home page two

Having issues aligning the title and description text on top of the image since they aren't in the same container?

@jmripper - isnt this on develop-for-jenny now?


Error: EPERM, device closed, operation not permitted

Hi. I try to read one photocell / sensor with MCP3008. Got this error code:
{ Error: ENOENT, No such file or directory errno: 2, code: 'ENOENT', syscall: 'open' }
/home/pi/tests/1/node_modules/mcp-spi-adc/mcp-spi-adc.js:109
  this._device.transfer(message, function (err, message) {
               ^

Error: EPERM, device closed, operation not permitted
    at AdcChannel.read (/home/pi/tests/1/node_modules/mcp-spi-adc/mcp-spi-adc.js:109:16)
    at Timeout._onTimeout (/home/pi/tests/1/index.js:32:15)
    at ontimeout (timers.js:458:11)
    at tryOnTimeout (timers.js:296:5)
    at Timer.listOnTimeout (timers.js:259:5)

My code:
var PhotoCell = mcpadc.open(5, {speedHz: 20000}, function (err) {
  if (err) console.log(err);

  setInterval(function () {
    PhotoCell.read(function (err, reading) {
      if (err) console.log(err);

      console.log((reading.value * 3.3 - 0.5) * 100);
    });
  }, 1000);
});

Wiring is ok - just like in readme. Node v9.5.0. Need some help please.

It looks like SPI hasn't been enabled. If a Raspberry Pi is being used, which appears to be the case, then SPI can be enabled with the raspi-config tool which can be called from the command line with the following command:
sudo raspi-config

Navigate to "5 Interfacing Options >> P4 SPI" and answer the question "Would you like the SPI interface to be enabled?" with "Yes", then answer "Ok", then answer "Finish".
If a fairly recent version Raspbian is being used a reboot should not be necessary. On an older version of Raspbian a reboot will be necessary.
After SPI has been enabled it should work.
This should be mentioned in the README.

Thanks for your straight answer!
I have enabled SPI through

sudo raspi-config

command and everything works great now.

That's good news 😄
I'll leave this issue open as a reminder that a note should be added to the README saying that SPI needs to be enabled before it can work.
One additional point, your program contains the following line of code:
console.log((reading.value * 3.3 - 0.5) * 100);

The formula (reading.value * 3.3 - 0.5) * 100 is for a TMP36 temperature sensor so it will need to be changed for a photocell.

Thanks for your advice. I will adapt the formula to work with the photocell.

The README was updated mentioning that SPI must be enabled to use mcp-spi-adc. See 33ed477e613987bf3f31df5b0aadcac3834a8692.

Sorry to re-open this, but I am using an MCP3004 and am getting this when trying to read channels 0, 1, and 2. When reading channel 0 and 1, there are no issues, but reading all 3 produces this error.

@amcelroy I can't reproduce the error. Please open a new issue containing the following information:

A description of the problem
The complete stack trace for the EPERM error
Minimal but complete sample code that can be used to reproduce the error


Ensure Token is ready for 2.10 or it's completely removed

Context
Parent issue: https://github.com/rancher/rancher/issues/47010
As part of the Imperative API, frameworks will be providing a dummy Token type that doesn't do anything (eg: no fields, etc) to help us develop the Imperative API and test it, and also to help kickstart the Token work for Collie.
If Token doesn't make it, we need to make sure that it's completely removed. This ticket tracks this and sets a release blocker to 2.10.

Since we haven't actually done https://github.com/rancher/rancher/issues/47009, there aren't any token types to remove for 2.10. I'm moving this to 2.11 milestone. (we might not even need it at all)
CC: @MKlimuszka

Closing as this is not needed anymore.


Change text screen IA

replace "ID document check" by "Real-time photo + ID document check"
position the IA on top of the list


can it use with iOS ? , i test it not working

No, there still no IOS implementation.

okay thank you, noted


Fix for raise condition

[https://github.com/IBM/fluent-forward-go/issues/21]

@BernhardSchmid I'm starting to think that the solution is to not guarantee that the code is safe for concurrency. The user can do a better job preventing concurrent operations that we can. It also simplifies our code and reduces the support burden.

previously approved then resubmitted


[mle] add discovery request callback

This commit allows notifying the application a discovery request is received, which is useful if the application wants to be more active when a joiner comes.

I reverted the url changes. @abtink @lanyuwen @jwhui


Action type: '@ngrx/store/update-reducers'

I'm faced with strange behaviour by @ngrx/store-devtools.
Environment:

Windows 10 Pro
Chrome Version 63.0.3239.132
Node v6.11.3, Node v8.9.4
Project generated with Angular CLI v1.5.0,
Angular v5.1.3

Problem
If I'm using this packages with corresponding versions:
    "@ngrx/effects": "^4.1.1",
    "@ngrx/store": "^4.1.1",
    "@ngrx/store-devtools": "^5.0.0",

When I'm opening Redux Devtools in Chrome among my Actions stack there is one with type @ngrx/store/update-reducers. That action causes the app to fail, cause the last action in stack has state as undefined. If I click 'skip' on that action (@ngrx/store/update-reducers) the app starts working normally and every action will have its proper state.
If I'm changing version to 4.0.0. everything works properly without any side actions.
    "@ngrx/effects": "^4.1.1",
    "@ngrx/store": "^4.1.1",
    "@ngrx/store-devtools": "4.0.0",

Could it be that one of your reducers is not returning the unmodified state for unknown actions in the default clause of an action switch?

@Halt001 For my case I have added a default clause for all reducers and got the issue with @ngrx/store/update-reducers
If you interested you can check the demo app where I fist faced with that issue: https://github.com/chekit/ngrx-101/tree/final/src/app/store/reducers (branch final)

I see what you mean. You don't have to skip @ngrx/store/update-reducers to make it work again. If you toggle skip on the last action in the stack the state is no longer undefined and it also starts working again.
Another observation is that when viewing the last action on the stack the state is undefined but the diff says states are equal, which isn't true. Also the last action in the stack is displayed in a slightly dimmed grey so it looks like some processing hasn't finished yet.


chore: release v1.10.0

:robot: I have created a release beep boop
1.10.0 (2024-07-22)
Features

Allow regex for include and exclude options (#148) (697b0d2)

Bug Fixes

Use correct format when resolving exports from relative paths (#145) (632802f)

This PR was generated with Release Please. See documentation.

:robot: Created releases:

import-in-the-middle-v1.10.0
:sunflower:


Suggestion: "-includeAL" should include source code from any published extension in the container

Would it be possible to have "-includeAL" also include any published/installed extension from the container?
Example: Microsoft has published some of their own danish extensions in the "-dk" images from mcr.microsoft.com, and if one is to extend these extensions from visual studio code, it would be nice to be able to see inside the code that is being extended.

-includeAL is only for baseapp
You can use extract-appfiletofolder if you want to extract app files to .al files.
When specifying a file name, you can specify :c:\applications\xxx\name.app (: before the name) to indicated that the file name refers to a name inside the container.

When executing from Powershell ISE:
Extract-AppFileToFolder -appFilename C:\Extensions\FIK.app -appFolder C:\Temp\FIK -generateAppJson
I get this error:
The variable '$binaryReader' cannot be retrieved because it has not been set.

If I am to repro this, I need to know more about what app it is, where you get it from, what version etc.

I am using this image: "mcr.microsoft.com/businesscentral/sandbox:14.5.35970.37991-dk"
The app comes preinstalled with the image.
The app is "Payment and Reconciliation Formats (DK)", version 14.5.35970.37991, App-ID 644f7e5b-a81c-408a-aaa2-766e655a80a3.
The app-file is named FIK.app and can be found in the folder C:\Extensions\

You need to copy the file out from the container.
Copy-FileFromBCContainer -containerName "test" -containerPath "C:\Extensions\FIK.app" -localPath "C:\ProgramData\NavContainerHelper\FIK.app"
Extract-AppFileToFolder -appFilename "C:\ProgramData\NavContainerHelper\FIK.app" -appFolder "C:\ProgramData\NavContainerHelper\FIK" -generateAppJson

Thanks

remember that apps are in different locations in 15.x and 14.x in the container (and on the DVD).
You can also download the app from the container by using Get-NavContainerApp - that is the same in 14.x and 15.x
Get-NavContainerAppInfo -containerName test -tenantSpecificProperties | % {
    if ($_.isinstalled) {
        Write-Host $_.name
        Get-NavContainerApp -containerName test -credential $credential -publisher $_.publisher -appName $_.name -appVersion $_.Version -appFile "c:\temp\$($_.name).app"
    }
}

Remember also that if apps have ShowMyCode = false - then you of course cannot extract the source code.


Overlord

changes based on Dan's feedback

Overall, this looks very nice. Good application code for your overlord.rb and bomb.rb files. Good unit tests overall (take a look at the spacing though) and good Cucumber tests. Take a look at writing your scenarios at a higher level of abstraction. This will help new people to a project understand how your app works, as well as business owners.
Your are good to go on this!


JsonString and deserializing a JSON string containing an array of objects

Am I missing something, or does using JsonString to deserialize a JSON string containing an array of objects not work?
use serde::{Deserialize, Serialize};
use serde_json::json;
use serde_with::serde_as;

#[test]
fn deserialize_json_array_str() {
    let value = json!([Inner { name: "One".into() }, Inner { name: "Two".into() }]);
    let value = json!({"value":value.to_string()});

    println!("Value being deserialized: {value:#?}");
    println!("JSON representation: {value}");

    let outer: Outer = serde_json::from_value(value).expect("Failed to deserialize");
    assert_eq!(
        vec![Inner { name: "One".into() }, Inner { name: "Two".into() }],
        outer.value
    );
}

#[derive(Debug, Deserialize)]
#[serde_as]
struct Outer {
    // These other two attempts made no apparent difference.
    // #[serde_as(as = "JsonString<Vec<Inner>>")]
    // #[serde_as(as = "JsonString<Vec<_>>")]
    #[serde_as(as = "JsonString")]
    pub value: Vec<Inner>,
}

#[derive(Debug, PartialEq, Eq, Serialize, Deserialize)]
pub struct Inner {
    pub name: String,
}

The output is:

Value being deserialized: Object {
    "value": String("[{\"name\":\"One\"},{\"name\":\"Two\"}]"),
}
JSON representation: {"value":"[{\"name\":\"One\"},{\"name\":\"Two\"}]"}
thread 'js::deserialize_json_array_str' panicked at src/main.rs:469:58:
Failed to deserialize: Error("invalid type: string \"[{\\\"name\\\":\\\"One\\\"},{\\\"name\\\":\\\"Two\\\"}]\", expected a sequence", line: 0, column: 0)

You need to switch the order of the macro annotations around. serde_as must always come before the derive. The reason is that the macro order is significant in Rust as the macros are expanded top down. You need to place serde_as first as it needs to run before the derive generates all its code, as otherwise the derive will not know about the changed behavior. That should fix everything. You now just need to import serde_with::json::JsonString.
#[serde_as]
#[derive(Debug, Deserialize)]


Feature Requests: Support for Live streams. RTMP/RTSP/HLS?

I would like to request support for live cams if possible.
I have a bridge for my security camera at home. It would be great if my computer monitor would serve as a security display when not being used for the computer.
https://github.com/mrlt8/docker-wyze-bridge

Hey @fbc
I'll have to have a look, biggest issue conceptually I have with this is that videos have a finite duration, and a camera kind of breaks that paradigm. I'll keep it in mind for next major version, as there's demand in general for more sources than just videos, but would require some important changes.

Just a quick +1 to this feature request! Exact same use-case, only I'm using Ubiquiti cameras.

I understand the need but this is soooo far from what Aerial does (playing fixed length videos).
I could see maybe adding an overlay that let's you put a camera in a corner, on top of an aerial playing.
But using a camera as source is such a specific case that I think a specific screensaver made for that purpose would better serve you two.

+1


dataset SEARCH17

I have a project follow your work.
I want to reproduce your results of Search personalization , but I cannot find the dataset SEARCH17.
Would that be possible to share this data?!
I would really appreciate if you can help me,

You should contact the second author to get the SEARCH17 dataset.

Hi, SEARCH17 is released as I get an agreement from the second author.


Enable/disable select elements

The directive does not affect select elements. This pull requests fixes it.

this change looks like the same as here, and I already merged that, probably I can close this?

looks like already implemented here


Adding get network status

Sometimes we just need to find out whether or not we're connected to a network!

This functionality is actually already provided by React Native.
https://facebook.github.io/react-native/docs/netinfo.html
Thanks though!

Yeah I tried that. It doesn't work when the app is backgrounded -- netinfo always returns the last status it received before the app is backgrounded, at least in Android
The only way for react native to get accurate status while the app is backgrounded is to make that call to the module... you can try this out to make sure I'm not nuts :)


Create a "TS basics" tut

This provides some definitions, etc and I think is a useful initial tutorial page, to be read after "what is a tree sequence" but before actually getting stuck in to python code and algorithms. It's largely taken from the BDI workshop notebook, hence closes #25

This is really good @hyanwong, just a couple of quibbles.

Cheers @benjeffery - all suggestions added and a few more tidies.

Thanks both. If we could merge this I can ping Konrad who is looking for material (I don't thing automerge is set up on the tutorials repo yet.


Testing multiple specs

Hi,
I am trying to run multiple specs that are defined in the config.js file.
specs: ['folder1/test1.spec.js', 'folder2/test1.spec.js', 'folder2/test2.spec.js']
When I run this, one browser window opens and it starts the first test, but then without finishing the first test, it will start the second.
Is there some option that makes sure that the test will start when the other test is finished? Or am I doing something wrong here?
Thanks in advance!

I have  to run two senerios using protractor. config.js I given two of the spec name. But not running second one. here is the sample. Please advice. Thanks
config.js
specs: ['mcac_jm_online_NRE_flow.js','mcac_jm_online_RTE_flow.js'],

Hi,
Is it possible to include the 2nd file in the end of 1st test file ?
Like with coding in asp or php we can transfer execution to other file can this be done with jasmine
specs: ['sendmsg.ts']
sendmsg.ts
describe(){
it (){
run case
}
call readmsg.ts;
}
Thanks


how to use hook to ref ActionSheet

<CustomActionSheet               ref={o => ActionSheet = o}               title={'Please select an option for uploading photo'}               options={['Choose Photo', 'Take Photo', 'Cancel']}               cancelButtonIndex={2}               onPress={(index) => {               }}             />
to show ActionSheet
showActionSheet = () => {     this.ActionSheet.show()   }

I just solve the problem. here is some step i followed.
`const CustomActionSheet = useRef();
<ActionSheet               ref={CustomActionSheet}               title={'Please select an option for uploading photo'}               options={['Choose Photo', 'Take Photo', 'Cancel']}               cancelButtonIndex={2}               onPress={(index) => {               }}             />
CustomActionSheet.current.show()


Android and iOS support?

The folder Plugins/iTwin/ThirdParty/iModelDecoder contains compiled libraries for windows and mac os, can iModelDecoder source files be complied for Android and iOS and is this on the Roadmap for development?

Hi Nicholas,
thank you very much for your feedback.
We're still evaluating support for both platforms.
Best regards,
Daniel


design problem

specially can guide or help me where i can do custome butotn design ui or javascript part with for example
i want to build ecommerce platform  online add to cart functionality to payment quantity add etc etc checkout etc etc possible iwth custome UI how i can do ?

see https://app.gitter.im/#/room/#tockchat_Lobby:gitter.im


Make Random Beacon test use contracts from deployment scripts

Currently, in the RandomBeacon we deploy contracts for test fixtures with ethers. In other projects, we use in tests the contracts that were deployed with hardhat deployment scripts. We need to update Random Beacon's tests to do the same as we do e.g. in ECDSA: https://github.com/keep-network/keep-core/blob/82cfa9e8305f46fc443cc4054111caf9f991529f/solidity/ecdsa/test/fixtures/index.ts#L67-L82

Tests were updated in the following PR https://github.com/keep-network/keep-core/pull/3018


chore: bump peer dependencies to match the versions from dev dependencies

Fixes #119
Please let me know if there was a valid reason for the versions to actually be different.

I don't think there is a reason. Thanks for the contribution, will release a new version in a moment.

v1.0.45 is published, could you please confirm that it works for you?

@mpashkovskiy Yes, this fixes it.


Validate at least 1 rector rule on Demo page

@staabm this should fix https://github.com/rectorphp/getrector-com/issues/2038
it is not perfect, but will make less notice

All checks have passed 🎉 @TomasVotruba it is ready for review.

ping @TomasVotruba
Was trying to validate a rule on the demo environment but encountered the error mentioned in https://github.com/rectorphp/getrector-com/issues/2038

Rebased 👍

Thank you :+1:


AM-1108- Update the return type of api instead of ResponseEntity<Obje…

JIRA link (if applicable)
https://tools.hmcts.net/jira/browse/AM-1108
Change description
Performance testing support
Does this PR introduce a breaking change? (check one with "x")
[ ] Yes
[ x] No

Purpose of this PR to change the  return type ResponseEntityof this service instead of ResponseEntity.
ORM is  getting the Map<String, Map<String,Map>> from RAS. So, it is a huge change and impact the existing logic.
So, I am closing this PR.


How is the number of rows for each embedding bag decided?

Hi,
From my understanding, in DLRM, the number of embedding bags is the same as the number of sparse feature types. For example, for Kaggle data,  there are 26 sparse feature types and thus 26 different embedding bags.
Now, I was wondering how the number of rows for each embedding bag is decided. Is the number of rows for an embedding bag random, or is it proportional to the data size or any other factors?
Thank you!

Your understanding about sparse features and embedding tables is correct.
In this dataset by default the number of rows in an embedding is based on the unique indices that a sparse feature can attain, so that each unique index maps to a single embedding vector. However, if the index range is too large, we can limit it with --max-ind-range parameter, the value of which is typically chosen empirically.

Got it. Thanks!


Build anomaly in generateProjections

I've encountered a build anomaly when building Users library. Here's a piece of log following ./gradlew clean build --info:
...
> Task :users-db:generateRejections UP-TO-DATE
Caching disabled for task ':users-db:generateRejections' because:
  Build cache is disabled
Skipping task ':users-db:generateRejections' as it is up-to-date.
:users-db:generateRejections (Thread[Daemon worker Thread 3,5,main]) completed. Took 0.0 secs.
:users-db:compileJava (Thread[Daemon worker Thread 3,5,main]) started.
...

The strange thing about this log is this line:
Skipping task ':users-db:generateRejections' as it is up-to-date.

It could not be so after clean.
Message types under Rejections where generated.
The directory generated/spine was not created.
The problem did not go away after I re-started the computer.
I know this sounds strange,  but it did disappear after I created another user account (admin one) and logged from my primary account out to test the new one. Following the request from @dmdashenkov to provide a link following ./gradlew clean build --scan, the build went successful.
I'm filing this issue to keep the records just in case the anomaly manifests again.

No longer relevant to this repository.


TextBox looks ugly when IsEnabled="false"

Any way to change the background of the textbox when it's disabled?

Agree! Disabled background should cover all the TextBox's region

The background of the textbox is transparent by default. It looks like you should modify the background of the container its inside.

I might work on this, I've already done a POC but couldn't get it working quite right.
The material design spec covers disabled text input fields:


Ignore downloaded or locally built files.

Add a basic .gitignore to ignore files that we downloaded or built locally. This makes graphical git tools such as Visual Studio Code much less cluttered.

Thank you for this, Jay.  I really know very little about git.
Bill


Allow setup-env to specify Python version

🗣 Description
This commit is introducing 2 new flags into the setup-env script. -l or--list-versions will list available Python versions and allow the user to select a version interactively. The second flag -v or --version will allow a user to set the version if installed. (e.g. ./setup-env -v 3.9.6)

💭 Motivation and context
This will resolve 153. The setup-env script currently creates the Python virtual environment without specifying a Python version via the command pyenv virtualenv <venv_name>. Because no Python version is specified, pyenv uses the system Python. This pull request will allow a user to specify which Python version they want to use.

🧪 Testing
Automated tests all passed. I tested the changes locally with different Python versions installed on my system.

✅ Pre-approval checklist

[x] This PR has an informative and human-readable title.
[x] Changes are limited to a single goal - eschew scope creep!
[x] All relevant type-of-change labels have been added.
[x] I have read the CONTRIBUTING document.
[x] These code changes follow cisagov code standards.
[x] All relevant repo and/or project documentation has been updated to reflect the changes in this PR.
[x] All new and existing tests pass.

I tested this script via

./setup-env
./setup-env -l
./setup-env -v 3.12.1 (Python 3.12.1 is installed on my system)
./setup-env -v 3.13.1 (Python 3.13.1 does not exist and hence is not installed on my system)
All these cases now appear to function as expected, but it would be good if someone else tested as well since I am not in the habit of using this script - although with these changes I intend to start.

There is one more problem. The original script allows one to specify the name of the virtual environment to be created, but this functionality is broken if you specify either of the new options.
To accomodate this, I suggest changing the option parsing code to use getopt. You can find lots of tutorials for getopt online.
Note that I also unchecked some of the checklist items in the PR description since they have not been done. If they do not need to be done then you can simply remove them.

Thanks for catching this I will get it fixed.

There is one more problem. The original script allows one to specify the name of the virtual environment to be created, but this functionality is broken if you specify either of the new options.
To accomodate this, I suggest changing the option parsing code to use getopt. You can find lots of tutorials for getopt online.
Note that I also unchecked some of the checklist items in the PR description since they have not been done. If they do not need to be done then you can simply remove them.

Thanks for catching this I will get it fixed.

If it's OK with the other members of @cisagov/team-ois, I have no issue with you changing the script to require a CLI switch (like -n/--name) when specifying the name of the Python virtual environment.  That might make things a bit easier.  I don't think that's a feature that gets used very often.

If it's OK with the other members of @cisagov/team-ois, I have no issue with you changing the script to require a CLI switch (like -n/--name) when specifying the name of the Python virtual environment. That might make things a bit easier. I don't think naming the virtual environment explicitly is a feature that gets used very often.

I'm fine with that since I always use the default name for the virtual environment (the repo name).

There is one more problem. The original script allows one to specify the name of the virtual environment to be created, but this functionality is broken if you specify either of the new options.
To accomodate this, I suggest changing the option parsing code to use getopt. You can find lots of tutorials for getopt online.
Note that I also unchecked some of the checklist items in the PR description since they have not been done. If they do not need to be done then you can simply remove them.

Thanks for catching this I will get it fixed.

Using getopt will probably give you the first bullet here for free.

@jsf9k @dav3r  I just made a decent amount of changes from my original script. But I do believe the intended functionality is now present.
Note the long options (--force) are now disabled because the BSD version of getopt doesn't support the long options. The GNU version does but then it would be a dependency requirement with running the ./setup-env script.
Also the -n flag is now present if a user wants to specify what name they want to use for a virtual environment and will default to the current directory if they don't.
I do think some of my changes could be improved upon but I wanted to get what I have working up to review to see if the functionality is what everyone was expecting.

My testing of these changes on Linux was successful.

Please update the PR description.  It mentions the new -l and -v options but not the new -n option.

Please update the PR description. It mentions the new -l and -v options but not the new -n option.

Updated! 😊


feat: Create PayButton API Route integrated with MySQL database

Description:

Adds a makefile, update REAMDE, fix docker-compose, add sequelize (to connect to mysql through NextJS) and required config, including initial models and migrations.
Adds integration with MySQL database for creating paybuttons through the API Route /api/paybutton

Test Plan: run yarn to update dependencies,  make dev to run the application and mysql, make init-db to run migrations and seeds. Every step should be successful.
Open up localhost:3000, then login and click on Add PayButton. On the browser console, should show a 201 response (successfully created)

As David suggested, moving this comment from Telegram to here

are you using sudo to run make?
i have a permission error i haven't bothered to track down yet when running make dev, but sudo make dev fixes it
theres a lot of different things happening with this branch, i'm not sure which change is the issue

Updates:

Removed init-db step, now make dev should build and run the scripts necessary to have the paybutton server working
Added instructions on how to get testing credentials to add to .env.local

Could you add testing credentials from supertoken to .env?
Also I'm still getting an error here

Just pushed a fix that only show social logins as long as you provide some credentials on .env file. (So it shouldn't break your app if you don't have it)
Let me know of any more issues, and if needed testing credentials, hit me up on telegram @Gattaca1

There are a ton of features in this branch, lets break it down some so its easier to deconstruct
Updating the README can happen in any branch to reflect changes, but just about everything else should have its own branch

Adds a makefile & update REAMDE,

fix docker-compose,

Dependency changes,

Add MySQL,

add sequelize (to connect to mysql through NextJS) and required config, including initial models and migrations.

Adds integration with MySQL database for creating paybuttons through the API Route /api/paybutton,

social login changes,


Feature request: Add a dispatch_loop script to rmf_demo_tasks

Expected behavior
A script that submits a loop task to the dispatcher node via the appropriate service call.

https://github.com/osrf/rmf_demos/pull/199


Added element sanitization. Added additional bundling process.

I added rudimentary HTML sanitation to the project, by embedding the minimal strip-tags npm module as commonJS. Because of the way Rollup seems to work, I also had to add an additional Makefile to the root directory. It will bundle the dependencies and create a file in dist called innerself.js.
Because of the condensing, I also ended up moving the logger functionality straight into the src/index.js file. It is now exported as an optional { logger }.
The additional module that I ended up creating in filter.js is run whenever the view is updated. Before the string is exported to the render, it is then sanitizied removing any script, iframe, or what have you.
I also included an optional argument to the sanitizer, so in the event you wish for there to be more specific abstraction (i.e only allowing inline elements in an input tag`) that can be done.
I am confident I updated the README.md to reflect these changes as well.
All in all, the rendering is now sanitized.

One thing to not. I did not add an index.jsto the root. I was not sure how you wanted that done, because of the re-bundling.

Another thing, I might fiddle around with the Rollup configuration to see if there is a nicer way to that code. Right now it seems it's bundling it as if it were ES5 code instead of ES2015.

So based on what you're saying, I'm considering removing the new Makefile, and instead embedding the sanitize functionality directly into the original index.js
This being said.... Because of your specific needs, it might be feasible to have two separate files, one which contains the small 400 byte library, and another that contains an embedded sanitize solution.
I'm a little confused about your anti-build requirements for the main js file, but at the same time understand. The library is nowhere large enough by itself to warrant using a build file.
Under these conditions, I am considering having a clone of index.js (likely named clean.js or something close). The issue I had with this approach is that the Rollup implementation in the example01 builder wouldn't recognize the import statements inside the root index.js. This can definitely be achieved by specifically doing something like this:
import html from `innerself/clean` // or `innerself/safe`
// .. other imports and code

I'll see what I can do.

@flynnham Doesn't https://github.com/stasm/innerself/commit/c4c023173b17abfe30f112b9cfcc4982b1b9e964 solve most of the current sanitization needs? I'm trying to udnerstand what's missing before we add more :)

I failed to see you already implemented that last night. My bad. I'll close the pull request.

Thanks again!
To close the loop on your previous comment: I think the general approach to offering a safer version of the library (or just the html helper in this case!) as innerself/safe was sound. If there are more features to add, putting them in separate modules is the way to go.


[nr-k8s-otel-collector] add documentation for user's personal collector config

Is this a new chart
What this PR does / why we need it:
Which issue this PR fixes
(optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged)

fixes #

Special notes for your reviewer:
Checklist
[Place an '[x]' (no spaces) in all applicable fields. Please remove unrelated fields.]

[ ] Chart Version bumped
[ ] Variables are documented in the README.md
[ ] Title of the PR starts with chart name (e.g. [mychartname])

@bpschmitt this PR may be of  your interest


Fixed URI encoding error in HTTP notify callbacks for URL values that have more than one querystring argument

Changed encoding of three query args that are themselves URLs (swfurl,
tcurl, and pageurl) that should be encoded as URI components (rather
than just ESCAPEd) to prevent double encoding errors. If NGX_ESCAPE_ARGS
is used to escape a swfurl value of
'http://example.com/?arg1=val1&arg2=val2'
the key/value pair of arg2/val2 would lose association with swfurl, and
will appear as a property or argument in the same context as swfurl,
tcurl, pageurl, etc. Encoding as a URI component ensures encapsulation
of this value isn't broken.

Failed to include identical changes to ngx_rtmp_netcall_module.c, but exact same issue.
See:
https://github.com/tsunami650/nginx-rtmp-module/commit/28a1a340a610e73bf4b39fde2796d0e10303cf43


To validate this pull request against other sbt modules, please visit this link.


[GOBBLIN-1716] refactor HighLevelConsumer to make consumer initializa…

allow SpecStoreChangeMonitor to initialize partition and offsets
create unique group_id for each host's consumer client monitor

Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA

[X] My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"

https://issues.apache.org/jira/browse/GOBBLIN-1716

Description

[X] Here are some details about my PR, including screenshots (if applicable):
Derived classes of HighLevelConsumer, for example SpecChangeMonitor, may want to initialize the kafka partitions and offset differently so we refactor the base class to grant this configurability. The default implementation remains the same as the original.

Tests

[X] My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

[X] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":

Subject is separated from body by a blank line
Subject is limited to 50 characters
Subject does not end with a period
Subject uses the imperative mood ("add", not "adding")
Body wraps at 72 characters
Body explains "what" and "why", not "how"

Codecov Report

Merging #3570 (86edbbd) into master (31f7ec5) will increase coverage by 3.94%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master    #3570      +/-   ##
============================================
+ Coverage     46.81%   50.75%   +3.94%
+ Complexity    10566     5799    -4767
============================================
  Files          2111     1061    -1050
  Lines         82468    40264   -42204
  Branches       9174     4507    -4667
============================================
- Hits          38607    20438   -18169
+ Misses        40301    18066   -22235
+ Partials       3560     1760    -1800

Impacted Files
Coverage Δ

...pache/gobblin/runtime/kafka/HighLevelConsumer.java
0.00% <0.00%> (ø)

...apache/gobblin/runtime/metrics/RuntimeMetrics.java
0.00% <ø> (ø)

...lin/data/management/copy/iceberg/IcebergTable.java
88.67% <0.00%> (-11.33%)
:arrow_down:

...n/data/management/copy/iceberg/IcebergDataset.java
83.56% <0.00%> (-0.73%)
:arrow_down:

...data/management/copy/RecursiveCopyableDataset.java
86.36% <0.00%> (ø)

...ta/management/copy/iceberg/IcebergHiveCatalog.java
0.00% <0.00%> (ø)

...management/copy/iceberg/IcebergCatalogFactory.java
0.00% <0.00%> (ø)

...anagement/copy/replication/ConfigBasedDataset.java
68.87% <0.00%> (ø)

...bblin/kafka/client/GobblinKafkaConsumerClient.java

...lin/service/monitoring/SpecStoreChangeMonitor.java

... and 1064 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


$key is not interpolated

https://github.com/sabre-io/cache/blob/042e4b24cea4347321739bc5470e78749bc27ffe/lib/Memcached.php#L81

Could you elaborate a bit?

Ah my mistake.
I thought the intention was to include the content of $key but I see now that you always want the literal string '$key must be a string'.

Edit: do you know why they didnt type hint string in the psr interface?

I guess not all cache backends necessarily use only strings as cache-keys.
will close the issue for now, feel free to ask further related questions


add new resource list page

docs project
adding a new page that explains all the available resource types dbt supports and also their supported properties/configs.
Resolves #4314
https://github.com/dbt-labs/docs.getdbt.com/issues/4314

@mirnawong1 Any status updates for this draft?

hey @matthewshaver , yes still working on through whenever i have a spare moment. it's been a bit busy so haven't had time to write this all up since we have product updates to prioritize


Inherited class does not load properly if used with tool

I see multiple issues related to class_name, but nothing obviously related to this one.
Godot 3.1.1 Stable
Windows 10
Issue description:
Adding tool before extends works if you use it like this:
extends "res://BaseClass.gd"
but not like this:
extends BaseClass

Steps to reproduce:
#base class at res://BaseClass.gd
extends Node2D
class_name BaseClass

enum GameResourceType{ GRAIN, WOOD, STONE }

(the base class is properly registered in Godot)
This does not work:
tool
extends BaseClass

export(GameResourceType) var resource_type:int = 0 setget _set_resource_type

func _set_resource_type(value):
	resource_type = value

And this works:
tool
extends "res://BaseClass.gd"

export(GameResourceType) var resource_type:int = 0 setget _set_resource_type

func _set_resource_type(value):
	resource_type = value

Minimal project
class_name_and_tool.zip

The BaseClass should be defined with tool.

Sure. That make sense.
But there is inconsistence in behavior if you extend base class like:
extends "res://BaseClass.gd"
or
extends BaseClass
Shouldn't both ways behave in the same way?


Add cargo workflow

Add cargo workflow for crate publish automation

Here is an example of using a specific version from the coredb repo for actions https://github.com/CoreDB-io/control-plane/blob/4785467bf9a92e607599bcb254d2cda03cff1566/.github/workflows/deploy.yml#L28-L33

Thanks, I'll give that a shot.


Fix authorization URL

From the docs:

Note: This is the only step that requires an endpoint on www.dropbox.com. All other API requests are done via api.dropboxapi.com, content.dropboxapi.com, or notify.dropboxapi.com.

Until recently it also worked with api.dropbox.com but this is seemingly no longer the case.

Yes. I need this update.


[Bazel CI] Unit tests failed in build

CI: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/4026#01910c21-7919-4709-b5e8-675f6fba3766
Platform: Ubuntu
Logs:
//base:unit_tests FAILED

1) getPendingExternalDeps_followJavaDeps_allBuilt(com.google.idea.blaze.base.qsync.DependencyTrackerImplTest)
value of: getPendingExternalDeps(...)
expected to be empty
but was : [@@rules_jvm_external++maven+com_google_guava_guava//jar:jar]
	at com.google.idea.blaze.base.qsync.DependencyTrackerImplTest.getPendingExternalDeps_followJavaDeps_allBuilt(DependencyTrackerImplTest.java:91)
2) getPendingExternalDeps_followJavaDeps_noneBuilt(com.google.idea.blaze.base.qsync.DependencyTrackerImplTest)
value of       : getPendingExternalDeps(...)
missing (1)    : @@com_google_guava_guava//jar:jar
unexpected (1) : [@@rules_jvm_external++maven+com_google_guava_guava//jar:jar]
---
expected       : [@@com_google_guava_guava//jar:jar]
testing whether: actual element is repository-mapped equal to expected element
but was        : [@@rules_jvm_external++maven+com_google_guava_guava//jar:jar]
	at com.google.idea.blaze.base.qsync.DependencyTrackerImplTest.getPendingExternalDeps_followJavaDeps_noneBuilt(DependencyTrackerImplTest.java:76)
FAILURES!!!
Tests run: 459,  Failures: 2

Culprit:
Steps:
git clone https://github.com/bazelbuild/intellij
git reset d4fcd51e0c7dcb94e22154f1d0a50d241b2eb6d3  --hard
export USE_BAZEL_VERSION=a7264d99e2d14463cb683a4091d5ad0e08431205
bazel test --define=ij_product=android-studio-latest //:aswb_tests

CC Greenteam @fweikert

Also seen in
Clion https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/4026#01910c21-6ced-4872-b8e5-44414a605cf9
Intellij : https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/4026#01910c21-70db-4bfc-9468-b6e1a31b6c79
IntelliJ Aspect : https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/4026#01910c21-722e-4aa5-9b72-5c0ba3161d06


Aruco board creation feature from 695

Implementation of requested feature #695

check intendation
comment the new API
squash all commits into one

@paroj done.. or not?

Replaced by #771


Incorrect version number in usage documentation

When I go to usage doc I read
Please ensure that you're using<EMAIL_ADDRESS>or higher
but the latest version for now is 0.81.3
Please fix or explain.

Thanks, I made PR with correct versions. Changes were merged earlier than I expect 🐑
Use 0.81.2 and higher.


Nothing shows up in Workers admin

Should my celery workers automatically show up in the Workers admin section? Nothing shows up there. Do I have to add them manually some how?

They should show up automatically. Are you running celerycam?

I don't believe I'm running celerycam

To be clear: this is for task execution monitoring in admin. If you don't want/need to see the state of running tasks in admin, it is safe to ignore all this.
First, you need to run all your workers with -E argument, so that it would send events about its state.
Then you need to tun python manage.py celerycam. This will start to fill information into Tasks and Workers admins.

Gotcha. Didn't see that anywhere in the docs. I'm already running Flower. I was just curious if the Workers/Tasks admin pages were any different than what I can see in Flower.


TypeError: can't pickle Environment objects when num_workers > 0 for LSUN

The program fails to create an iterator for a DataLoader object when the used dataset is LSUN and the amount of workers is greater than zero. I do not have such an error when work with other datasets. Something tells me that the issue might be caused by lmdb. I run on Windows 10, CUDA 10.
Code:
import torch.utils.data
import torchvision.datasets as dset
import torchvision.transforms as transforms

dataset = dset.LSUN(root='D:/bedroom_train_lmdb', classes=['bedroom_train'],
                            transform=transforms.Compose([
                                transforms.Resize((64, 64)),
                                transforms.ToTensor(),
                                transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5)),
                            ]))

dataloader = torch.utils.data.DataLoader(dataset, batch_size=128,
                                             shuffle=True, num_workers=4)

for data in dataloader:
    print(data)

Error:
Traceback (most recent call last):
  File "C:/Users/x/.PyCharm2018.3/config/scratches/scratch.py", line 15, in <module>
    for data in dataloader:
  File "C:\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 819, in __iter__
    return _DataLoaderIter(self)
  File "C:\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 560, in __init__
    w.start()
  File "C:\Anaconda3\lib\multiprocessing\process.py", line 112, in start
    self._popen = self._Popen(self)
  File "C:\Anaconda3\lib\multiprocessing\context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "C:\Anaconda3\lib\multiprocessing\context.py", line 322, in _Popen
    return Popen(process_obj)
  File "C:\Anaconda3\lib\multiprocessing\popen_spawn_win32.py", line 65, in __init__
    reduction.dump(process_obj, to_child)
  File "C:\Anaconda3\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: can't pickle Environment objects

This seems to be a Windows-specific issue.
But note that even if we address this particular issue (I have no idea how to do it though), you would probably hit another issue further on, which is https://github.com/pytorch/vision/issues/619

this issue also appear in linux,  the reason is the opened lmdb env can not be pickled

@Santiago810 Do you know how to diagnose the issue of an un-pickleable lmdb env?

I have the same issue with dataloader when I do not use lmdb dataset.

I think this is a limitation of LMDB in python (and LSUN which uses LMDB internally), and I think there is not much we can do on torchvision side unfortunately.

I implemented my own LMDB dataset and had the same issue when using LMDB with num_workers > 0 and torch multiprocessing set to spawn.
It is very similar to this project's LSUN implementation, in my case the issue was with this line:
https://github.com/pytorch/vision/blob/master/torchvision/datasets/lsun.py#L18
When set to fork it works fine, but when using spawn it seems to try to pickle the dataset object which has the self.env attribute which is a lmdb Environment.
Just use it and discard the reference in the init then instantiate it again in the getitem and save the reference in the class.

@4knahs if you think you could send a PR fixing the LSUN implementation it would be great!

I saw a solution somewhere else by adding  getstate and setstate.
    def __getstate__(self):
        state = self.__dict__
        if self.db_type == 'lmdb':
            state["db_txn"] = None
        return state

    def __setstate__(self, state):
        self.__dict__ = state
        env = lmdb.open(self.db_path, subdir=os.path.isdir(self.db_path),
                                readonly=True, lock=False,
                                readahead=False, meminit=False,
                                map_size=1099511627776 * 2,)
        self.db_txn = env.begin(write=False)

This also doens't save self.env but instead of saving the txn.

Solution: open lmdb in worker_init_fn of torch.utils.data.DataLoader

Could you elaborate or give an example @Santiago810 ?

A possible solution is similar to the one for HDF5:

Do not open lmdb inside __init__
Open the lmdb at the first data iteration.

Here is an illustration:
class DataLoader(torch.utils.data.Dataset):
    def __init__(self):
        """do not open lmdb here!!"""

    def open_lmdb(self):
         self.env = lmdb.open(self.lmdb_dir, readonly=True, create=False)
         self.txn = self.env.begin(buffers=True)

    def __getitem__(self, item: int):
        if not hasattr(self, 'txn'):
            self.open_lmdb()
        """
        Then do anything you want with env/txn here.
        """

Explanation
The multi-processing actually happens when you create the data iterator (e.g., when calling for datum in dataloader:):
https://github.com/pytorch/pytorch/blob/461014d54b3981c8fa6617f90ff7b7df51ab1e85/torch/utils/data/dataloader.py#L712-L720
In short, it would create multiple processes which "copy" the state of the current process. This copy involves a pickle of the LMDB's Env thus causes an issue. In our solution, we open it at the first data iteration and the opened lmdb file object would be dedicated to each subprocess.

Thank you @airsplay . Excellent solution. You just saved me about a months work !!!

A possible solution is similar to the one for HDF5:

Do not open lmdb inside __init__
Open the lmdb at the first data iteration.

Here is an illustration:
class DataLoader(torch.utils.data.Dataset):
    def __init__(self):
        """do not open lmdb here!!"""

    def open_lmdb(self):
         self.env = lmdb.open(self.lmdb_dir, readonly=True, create=False)
         self.txn = self.env.begin(buffers=True)

    def __getitem__(self, item: int):
        if not hasattr(self, 'txn'):
            self.open_lmdb()
        """
        Then do anything you want with env/txn here.
        """

Explanation
The multi-processing actually happens when you create the data iterator (e.g., when calling for datum in dataloader:):
https://github.com/pytorch/pytorch/blob/461014d54b3981c8fa6617f90ff7b7df51ab1e85/torch/utils/data/dataloader.py#L712-L720
In short, it would create multiple processes which "copy" the state of the current process. This copy involves a pickle of the LMDB's Env thus causes an issue. In our solution, we open it at the first data iteration and the opened lmdb file object would be dedicated to each subprocess.

Hi airsplay,
This solution works fine, however I'm struggling to find a way to set the self.size property on the Dataset without loading it in the __init__ function beforehand. I cannot instantiate the torch.utils.data.DataLoader without making sure that __len__ returns a valid value. Right now I save the number of samples in a meta data file and load that manually, but is there a smarter way to do this?

@thecml you can open an LMDB environment in __init__ just be sure to close it within __init__

@thecml you can open an LMDB environment in __init__ just be sure to close it within __init__. So open it, assign a size variable which is called by __len__ and the close it

Excellent, thanks!

A possible solution is similar to the one for HDF5:

Do not open lmdb inside __init__
Open the lmdb at the first data iteration.

Here is an illustration:
class DataLoader(torch.utils.data.Dataset):
    def __init__(self):
        """do not open lmdb here!!"""

    def open_lmdb(self):
         self.env = lmdb.open(self.lmdb_dir, readonly=True, create=False)
         self.txn = self.env.begin(buffers=True)

    def __getitem__(self, item: int):
        if not hasattr(self, 'txn'):
            self.open_lmdb()
        """
        Then do anything you want with env/txn here.
        """

Explanation
The multi-processing actually happens when you create the data iterator (e.g., when calling for datum in dataloader:):
https://github.com/pytorch/pytorch/blob/461014d54b3981c8fa6617f90ff7b7df51ab1e85/torch/utils/data/dataloader.py#L712-L720
In short, it would create multiple processes which "copy" the state of the current process. This copy involves a pickle of the LMDB's Env thus causes an issue. In our solution, we open it at the first data iteration and the opened lmdb file object would be dedicated to each subprocess.

Hi airsplay,
This solution works fine, however I'm struggling to find a way to set the self.size property on the Dataset without loading it in the __init__ function beforehand. I cannot instantiate the torch.utils.data.DataLoader without making sure that __len__ returns a valid value. Right now I save the number of samples in a meta data file and load that manually, but is there a smarter way to do this?

works for 1 GPU with num_workers > 0, but failed for multi GPU


bug: portforward view too wide

In pods view:
press gp on a row.

The original issue is solved by #296 but found another issue on the gP (portforwards view). The size fluctuates when opening with empty content.

fixed with #4e6a94c


Increment versions for qumulo releases

Increment package versions for qumulo releases

API change check
API changes are not detected in this pull request.


ImportError: type object 'Distribution' has no attribute '_finalize_feature_opts'

Hello,
I encounter the following problem after using the command: pip3 install -e ".[test]" -vvv. with setuptools 46.0.0. It seems very new because I did not have the problem on the same code 15 days ago, and also because there is nothing containing this error message when doing a web search. Reverting to an older version of setuptools (like 42.0.0, or 45.3.0) make the problem disapear.
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-ephem-wheel-cache-j18ms5xi
Created temporary directory: /tmp/pip-req-tracker-ii4rcxhn
Initialized build tracking at /tmp/pip-req-tracker-ii4rcxhn
Created build tracker: /tmp/pip-req-tracker-ii4rcxhn
Entered build tracker: /tmp/pip-req-tracker-ii4rcxhn
Created temporary directory: /tmp/pip-install-nw20_kz_
Obtaining file:///home/pierre/PycharmProjects/project-name
  Added file:///home/pierre/PycharmProjects/project-name to build tracker '/tmp/pip-req-tracker-ii4rcxhn'
    Running setup.py (path:/home/pierre/PycharmProjects/project-name/setup.py) egg_info for package from file:///home/pierre/PycharmProjects/project-name
    Running command python setup.py egg_info
    Traceback (most recent call last):
      File "/home/pierre/PycharmProjects/project-name/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2454, in resolve
        return functools.reduce(getattr, self.attrs, module)
    AttributeError: type object 'Distribution' has no attribute '_finalize_feature_opts'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/pierre/PycharmProjects/project-name/setup.py", line 56, in <module>
        zip_safe=True,
      File "/home/pierre/PycharmProjects/project-name/venv/lib/python3.6/site-packages/setuptools/__init__.py", line 144, in setup
        return distutils.core.setup(**attrs)
      File "/usr/lib/python3.6/distutils/core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
      File "/home/pierre/PycharmProjects/project-name/venv/lib/python3.6/site-packages/setuptools/dist.py", line 425, in __init__
        k: v for k, v in attrs.items()
      File "/usr/lib/python3.6/distutils/dist.py", line 281, in __init__
        self.finalize_options()
      File "/home/pierre/PycharmProjects/project-name/venv/lib/python3.6/site-packages/setuptools/dist.py", line 706, in finalize_options
        ep.load()(self)
      File "/home/pierre/PycharmProjects/project-name/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2446, in load
        return self.resolve()
      File "/home/pierre/PycharmProjects/project-name/venv/lib/python3.6/site-packages/pkg_resources/__init__.py", line 2456, in resolve
        raise ImportError(str(exc))
    ImportError: type object 'Distribution' has no attribute '_finalize_feature_opts'
Cleaning up...
Removed file:///home/pierre/PycharmProjects/project-name from build tracker '/tmp/pip-req-tracker-ii4rcxhn'
Removed build tracker: '/tmp/pip-req-tracker-ii4rcxhn'
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Here is part of my setup.py:
TEST_REQUIRES = ["pytest-cov"]
SETUPTOOLS = "setuptools~=42.2.0"

setuptools.setup(
    entry_points={
        "console_scripts": [
            "scriptname=namespace.file:run",
        ]
    },
    packages=setuptools.find_namespace_packages(),
    package_dir={},
    install_requires=[
        "beautifulsoup4",
        "fluidtopics>=0.2.0",
        "mammoth~=1.4.10",
        "lxml",
        "pypandoc==1.4",
    ],
    setup_requires=[SETUPTOOLS],
    tests_require=TEST_REQUIRES,
    extras_require={"test": TEST_REQUIRES, "setup": [SETUPTOOLS]},
    zip_safe=True,
)

The problem came from the setup_requires=["setuptools~=42.2.0"], when using "setuptools~=46.0.0" locally. You can replace setup_requires=["setuptools~=42.2.0"], by setup_requires=["setuptools~=46.0.0"],.


test/oidc: rename TestPublisher, mark as abstract

Locally, this appears to suppress the warnings that @miketheman noticed in CI. Let's see what happens in CI now 🙂

Confirmed that the OIDC warnings are no longer present on CI as well.


ns register response says "mailto"

:NickServ!NickServ@localhost NOTICE 1b5c7a9ab190 :Account created, pending verification; verification code has been sent to<EMAIL_ADDRESS>Let's strip the mailto.

thanks @eklitzke for reporting this, I apparently became blind to it over the years


GPU memory leakage (CUDA out of memory after 2 epochs)

Hi,
Thank you for this repo.
Unfortunately, I am unable to train this repo for more than ~2 epochs, at the end I keep getting CUDA out of memory.
Tracking the GPU memory consumption a slow and steady increase is visible.
To check it, I have decreased the BATCH_SIZE to 6 while running on 3 RTX-2080 (10.7 g) GPUs, using nvidia docker cuda:10.1-cudnn8-devel-ubuntu18.04 and pytorch 1.8 (cuda 10.1).
Running code:
python train.py --gpu 0,1,2 --workdir /temp_out --model sparenet

Have you encountered such a problem? Is some variable history not released?

I might have spotted the problem, not sure as I have only briefly examined the code, but it seems that the actual losses (not the value or a detached version) are accumulating for visualization, will report here if this is the problem.

It seems that I was right, and I would be happy to get an approval here.
All the update operations should take items not Variables.

@orenkatzir
It makes full sense to me. Thanks for reporting the issue! We will fix it.


Refactor #32899 JsonResponse docs for safe parameter.

update the documention of JsonResponse for safe parameter.

@hammadarshad1 Thanks for this patch, however this ticket is assigned to Hasan who already submitted PR #14598.


[Improve][CDC] Bump the version of debezium to 1.9.8.Final

Purpose of this pull request

Bump the version of debezium to 1.9.8.Final for issue #6708.
Does this PR introduce any user-facing change?

How was this patch tested?

Check list

[ ] If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
[ ] If necessary, please update the documentation to describe the new feature. https://github.com/apache/seatunnel/tree/dev/docs
[ ] If you are contributing the connector code, please check that the following files are updated:

Update change log that in connector document. For more details you can refer to connector-v2
Update plugin-mapping.properties and add new connector information in it
Update the pom file of seatunnel-dist

[ ] Update the release-note.

Please update the link about debezium in MySQL-CDC.md.

[Doc] Fix dead link with debezium #6283

done

good job

cc @EricJoy2048

good job

Thanks for your contribution. This is a good feature.


Some item uploads with no listings fail to be persisted

Universalis: https://discord.com/channels/616826096897228801/616826096897228805/1277335589556977694
Dalamud: https://discord.com/channels/581875019861328007/1084906665343729725/1277413587182157885
Items with 0 listings seem to be ignored, either in the client or the API. This has been fixed in the API as of #1130, and seems unlikely to be an API-level regression as not all clients are affected.
Affected upload clients:

ACT: ?
Standalone: No
Dalamud: Yes
Teamcraft: ?

Download
https://www.dropbox.com/scl/fi/ku9a1wblqyb84rb8ekase/fix.zip?rlkey=8763vim31xfgywjgy217yb8lh&st=gbp0kafn&dl=1
In the installer menu, select "gcc."


solid_dmft parser

And changes to adapt new metainfo in w2dynamics.

Hi @ladinesa ,
I was working on a new DMFT parser. This is linked with a new branch in nomad (https://gitlab.mpcdf.mpg.de/nomad-lab/nomad-FAIR/-/commits/1264-add-solid_dmft-parser). Some quantities (specially for Greens Functions) have been slightly modified to address multi-site calculations (i.e. now they have an extra dimension).
I thus had to modify the w2dynamics parser too to adapt to this changes.
Could you please review it? No rush, as I am still waiting for the first DMFT branch (1156) to be merge in nomad.
Also, I am not fully happy with the 'parse_scc' method. Maybe you can take a closer look and give me suggestions on how to optimize it :slightly_smiling_face:

great. please merge.


feat(core): support nested sub menu

Description
Now we're able to add sub menu for menu items, even nested sub menus.
MenuGroup -> [MenuItem, MenuItem, MenuItem]

MenuItem -> *MenuGroup(optional)

Customizability improved! 🎉

To implement:

[ ] configurable visible trigger (click, hover...).
[ ] hideOn: hover | click, parent first.


Alpha release

Implements keypoint detection app with the following capabilities:

Keypoint detection on image files with logging
Keypoint detection on camera snapshots in soft-realtime

Seems like we won't merge this


Test PR for build report in comment

Docs Build status updates of commit 097d7d4:
:warning: Validation status: warnings

File
Status
Preview URL
Details

test-repo-280322-01/index.md
:warning:Warning
View
Details

test-repo-280322-01/index.md

Line 1, Column 1: [Warning: title-missing - See documentation] Missing required attribute: 'title'. Add a title string to show in search engine results.
Line 1, Column 1: [Warning: author-missing - See documentation] Missing required attribute: 'author'. Add the current author's GitHub ID.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 7cd41d7:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error: PullRequestMergeConflict] Cannot sync git repo to specified state. Please make sure the pull request is mergeable.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit 3706fea:
:warning: Validation status: warnings

File
Status
Preview URL
Details

test-repo-280322-01/index.md
:warning:Warning
View
Details

test-repo-280322-01/TOC.md
:bulb:Suggestion
View
Details

test-repo-280322-01/index.md

Line 1, Column 1: [Warning: title-missing - See documentation] Missing required attribute: 'title'. Add a title string to show in search engine results.
Line 1, Column 1: [Warning: author-missing - See documentation] Missing required attribute: 'author'. Add the current author's GitHub ID.
Line 5, Column 1: [Warning: h1-not-first - See documentation] Markdown content is not allowed before H1 'Welcome to test-repo-280322-01!'.
Line 0, Column 0: [Suggestion: toc-missing] Article is not referenced from a table of contents. Every article must be in a TOC.

test-repo-280322-01/TOC.md

Line 0, Column 0: [Suggestion: toc-deprecated] Markdown TOC files are deprecated. Convert to a YAML TOC.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


Use pip_system_certs in place of certifi

Related command
This is related to any TLS verification by az cli.
Is your feature request related to a problem? Please describe.
Managing certificates is very challenging in a corporate environment, the advice in https://learn.microsoft.com/en-us/cli/azure/use-cli-effectively?tabs=bash%2Cbash2#work-behind-a-proxy works, but is rather cumbersome.
Describe the solution you'd like
Using pip_system_certs would likely greatly improve the user experience by using the system certificate store, and likely reduce support requests etc related to problem like this.... just searching the issue queue with is:issue is:open certificate it looks like there are 105 at the time of writing.
See: pip_system_certs

Thank you for opening this issue, we will look into it.

This issue has been tracked by the underlying requests library: https://github.com/psf/requests/issues/2966

It seems that it does not support MacOS: https://gitlab.com/alelec/pip-system-certs/-/issues/19

Duplicate of https://github.com/Azure/azure-cli/issues/19305


Update Bolt version selector to optionally open in a new tab

Jira
N/A
Summary
Updates the new Bolt version selector on the docs site to optionally allow the Bolt version being selected to open in a new tab if the ⌘ (CMD) / meta key is being pressed down at the same time.
This is a small usability improvement to more easily allow multiple / different versions of Bolt to easily get pulled up side-by-side for comparison.
How to test
Bolt Version Selected Opens In A New Tab
Navigate to the updated Bolt Design System site from this PR branch (link will be below in the comments once this deploys) and try clicking on a version of Bolt from the dropdown menu while holding down the ⌘ key on your keyboard.
Confirm this opens in a new tab as expected.

Bolt Version Selected Opens In The Same Tab
Navigate to the updated Bolt Design System site from this PR branch (link will be below in the comments once this deploys) and try clicking on a version of Bolt from the dropdown menu without holding down any keys on your keyboard.
Confirm the version selected is navigated to in the current tab open (same default behavior on the current https://master.boltdesignsystem.com site)
CC @remydenton @margoromo @mikemai2awesome

Actually, that's just standard browser behavior... nevermind about the docs, that seems overkill.

Sweet! It'd be cool if there was some way for someone who hadn't seen this PR to know that it worked that way. Maybe a tooltip or something? That's not a reason to hold off on merging, just a nice to have.

Yep, I was thinking the EXACT same thing!
It could be an icon or something... I like the idea of tooltip support too... that'd be super helpful in a bunch of places on the docs site / Pattern Lab UI. We're getting there!

Actually, that's just standard browser behavior... nevermind about the docs, that seems overkill.

Hold down CMD key, have a small subtle icon show up on the right side of each dropdown item?

Hold down CMD key, have a small subtle icon show up on the inside of each dropdown item (on the right side)?

Cool!  Although, it would't actually teach users who didn't already know about the CMD key a new trick, and for users who do know that trick already it would be unnecessary UI candy.
We're definitely bike shedding here, by the way :).

Just to hop on the bikeshedding, if you support command + click, you should also support middle click, since they are generally interpreted the same across OSs.

Just to hop on the bikeshedding, if you support command + click, you should also support middle click, since they are generally interpreted the same across OSs.

@charginghawk I'm gonna see if we can slide this in middle click behavior (which I totally agree with) with the next iteration of this new component so we can still merge this in with the next v2.2.0 release ASAP but also get this behavior figured out too so other Bolt UI can benefit from this.


Fix: Updated UpdateCenterPublishedPluginDetectionScoring key

Description

Submitter checklist

[x] If the issue exists, it is well described and linked in the description
[ ] The description of this pull request is detailed and explain why this pull request is needed
[x] The changeset is on a specific branch

feature/ for new feature, or improvements
fix/ for bug fixes
docs/ for any documentation changes

[x] If required, the documentation has been updated
[ ] There is automated tests to cover the code change / addition

If there is no test, include an explanation why in the description

[x] Run mvn verify locally and all tests are passing successfully
[x] There is no new warnings on the code

Updated the scoring key value.

Hi @Jagrutiti Thanks for the code contribution.
It seems like that for io.jenkins.pluginhealth.scoring.scores.UpdateCenterPublishedPluginDetectionScoringTest.shouldScoreCorrectlyWhenPluginInUpdateCenter the program was expecting some very specific syntax:
expected: "update-center-plugin-publication-probe"
 but was: "update-center-plugin-publication"

Likewise the same for io.jenkins.pluginhealth.scoring.scores.UpdateCenterPublishedPluginDetectionScoringTest.shouldBadlyScorePluginWithNoProbe and io.jenkins.pluginhealth.scoring.scores.UpdateCenterPublishedPluginDetectionScoringTest.shouldScoreBadlyWhenPluginNotInUpdateCenter.
It would be great if you could address this by testing locally with mvn verify and checking no checks fail before commtting again.


Fixed the visibility for the default constructor of classes LocalRepl…

…icationImpl and RemoteReplicationImpl
See this comment.

Thank you, however it looks like it's still not sufficient. Following your doc snippet, you don't have any field for your Remote(Local)ReplicationImpl() so you cannot define them. Moreover artifactory.repository(<repo name>).replications doesn't exist.

I just opened #155 to make it work


m_jconf error in the linux terminal

STAT: include config: julius.jconf
ERROR: m_jconf: failed to open jconf file: julius.jconf
Try `-help' for more information.
how can i fix this error

I just ran into the same problem today and spent several hours trying to figure it out, not sure if this is the correct solution but it seemed to fix the m_jcomf error.
First of all, make sure you have all the files extracted from the zipped ENVR-v5.4.DNN.Bin.zip from the sourceforge. I noticed that I was missing the last couple of files from the zip, something just extracted weird. This included the julius.config file.
The second thing I did was put the Julius folder inside the ENVR-v5.4.Dnn.Bin folder. This allows the computer to grab both the ./julius/julius/julius and the julius.jconf files, if you run the  % ../julius/julius/julius -C julius.jconf -dnnconf dnn.jconf  command from the ENVR-v5.4.Dnn.Bin. directory (the same location you are in a the end of step 2)


Automated cherry pick of #3231: fix: 避免删除azure失败

Cherry pick of #3231 on release/2.12.
#3231: fix: 避免删除azure失败

/lgtm

/approve


Cosmetic issue: Help / usage text for kSNP4 references self as kSNP3.

When run with the command 'kSNP4 -h' (with -h not being a valid argument), the errors listed below occur, and then the usage statement is shown with the name of the program showing kSNP3 instead of kSNP4.
# kSNP4 -h
Error: Unrecognized argument (no valid argument: -h)
Error: Must specify -k, -outdir, and -in.  At least one is missing.
Usage: kSNP3 -k <kmer length> -outdir <output directory> -in <input fasta file> [<optional arguments>...]

Hi Jeremiiah,
I think your message is VERY important.
Your message suggests that we need to start a “to do” list for the kSNP4.1 release.  There were a number of your comments for the User Guide to which I responded something like “not now.  Perhaps in the next release”.  Those should all be added to the “to do” list.
My first candidate for the list is that we need to have a uniform way to display a help screen and that should apply to EVERY program that can be called directly by the user.  Entering the option -h should in every case display the help.  At this time some programs display help when no argument is entered.  Bad.  Not uniform.
For kSNP4 entering kSNP4 -h should display this:
Must specify -k, -outdir, and -in.
Usage: kSNP4 -k <kmer length, an odd number >= 7> -outdir  -in  […]
Example: kSNP4 -in Eco.in -k 21 -outdir EcoRun1
For optional arguments see the kSNP4 User Guide
Arguments may be entered in any order
Notice that under Usage -in does not mention fasta.  That is because the input file is not a fasta
Just below Usage is an Example line.  That is important because we cannot expect the user to know that  means “enter the name of the output directory here.  We both know that convention, but we can’t expect the user to know it.  The Example line is an explicit example that also serves to teach the meaning of what is on the Usage line.  The next line is because we don’t intend for the help screen to be a substitute for the User Guide.  The last line, which should be at the end of every help screen.
Notice also that there is no mention of an error because -h is a legitimate argument. Notice that I changed kSNP3 to kSNP4.
If, on the other hand, the user fails to enter one of the required arguments the Error screen will be displayed:
Error: Must specify -k, -outdir, and -in. At least one is missing.
Usage: kSNP4 -k  -outdir  -in  [...]
Example: kSNP4 -in Eco.in -k 21 -outdir EcoRun1
For optional arguments see the kSNP4 User Guide
Arguments may be entered in any order
My second candidate for the list is that EVERY argument should include a flag, even if there is a single argument.  The user should not have to remember which program command lines use a flag and which do not. I know that I have a hard time remembering which arguments MakekSNP4infile requires.  Just being able to enter MakekSNP4infile -h would save me from having to go to the User Guide again, and again, and again.  We need to keep in mind that most people won’t run kSNP4 or its utility programs very often and KNOWING that they can just enter -h to get a help screen will make our programs much more use friendly
Let’s start building the to do list separately, then reconcile our lists every week or so.
Barry
Barry G. Hall
Director, Bellingham Research Institute
@.***
http://www.bellinghamresearch.com http://www.bellinghamresearch.com/

On Oct 27, 2022, at 4:02 PM, kissake @.***> wrote:
When run with the command 'kSNP4 -h' (with -h not being a valid argument), the errors listed below occur, and then the usage statement is shown with the name of the program showing kSNP3 instead of kSNP4.
kSNP4 -h
Error: Unrecognized argument (no valid argument: -h)
Error: Must specify -k, -outdir, and -in. At least one is missing.
Usage: kSNP3 -k  -outdir  -in  [...]
—
Reply to this email directly, view it on GitHub https://github.com/kissake/kSNP4/issues/1, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEEW7EQG754QEGZVVHLF4PDWFMCXDANCNFSM6AAAAAARQQ7TTI.
You are receiving this because you are subscribed to this thread.


Weird locking

Firstly, a great idea to port the big number library for microcontrollers!
I uploaded the examples program sine.pde on an Arduino Uno clone through Arduino IDE 1.8.5 but I get weird locks of the Serial output. The serial console consistently produces only:
sin(0) = 0.00000000000000000000000000000000000000000000000000
sin(1) = 0.01745240643728351281941897851631619247225272030713
sin(2) = 0.03489949670250097164599518162533293735482457604328
sin(3) =

I thought that I could print the time when each iteration begins, and oddly enough, the loop reached 30 degrees before locking again.
Printing the iteration index of the sine routine shows that it always would reach precision = 19 and then just dies:
1.00000000000000000000000000000000000000000000000000
0.49999999999999999999999999999999999999999999999981
Time: 21665 ms  sin(31) = 30
29.00000000000000000000000000000000000000000000000000
28.00000000000000000000000000000000000000000000000000
27.00000000000000000000000000000000000000000000000000
26.00000000000000000000000000000000000000000000000000
25.00000000000000000000000000000000000000000000000000
24.00000000000000000000000000000000000000000000000000
23.00000000000000000000000000000000000000000000000000
22.00000000000000000000000000000000000000000000000000
21.00000000000000000000000000000000000000000000000000
20.00000000000000000000000000000000000000000000000000
19.00000000000000000000000000000000000000000000000000

Do you have any suggestion what could be causing this behavior? I have access to an oscilloscope but I am not experienced with it. Can it be useful for debugging? Thanks.

The documentation for the library here mentions:

Each digit requires a byte of memory, plus some overhead (eg. where the decimal point is) - large numbers will soon gobble up the available memory.

The later versions of the IDE must have slightly more overhead than when I tested on an earlier version. If you change the number of decimal places like this:
  BigNumber::begin (45);

Then it works up to sin(90).
As a general rule with this library, if it locks up try reducing the precision. We are using a processor with only 2 k of RAM.

Indeed, thank you for your suggestion. Even reducing the precision to 49 decimal digits sorts out the problem. Also, I tested compiling with -O2 instead of -Os and the additional optimisation made it possible to run the code with 60-digit precision. It's not that I truly need that many digits for my own project but I just wanted to find out what is going on.
For the sake of book keeping (50 digits):
-Os:
Sketch uses 10248 bytes (31%) of program storage space. Maximum is 32256 bytes.
Global variables use 380 bytes (18%) of dynamic memory, leaving 1668 bytes for local variables. Maximum is 2048 bytes.
-O2:
Sketch uses 12036 bytes (37%) of program storage space. Maximum is 32256 bytes.
Global variables use 380 bytes (18%) of dynamic memory, leaving 1668 bytes for local variables. Maximum is 2048 bytes.
I am now closing the issue.


Error throwing for missing arguments breaks in webapp2

basically, the PaleBaseResponse inherited by the APIError returns a tuple that works fine for flask, but needs to be run through the response handler for webapp2 in the same way that the actual content response is built on line 287-ish of pale/endpoint.py
Filing this as a TODO note for myself, since i'm not going to fix it right now.

#8 resolves the symptom (missing arguments no longer cause 500 errors), but the way the response is run through the handler is pretty terrible.  Closing for now, pending some general refactoring in endpoint.py