(post comments) Intoduction to Cobol to Hive Serde blogs

Testing

Hello rbheemana,
I was following this blog since a while and wanted to try in our system, however  i have just noticed that there is no JAR file available in the target folder, may I know where can i find and how I can try it?

Hi,
However I got some jar file in your activity folder, I have tried adding the jar and create the table, I am getting the following error....
Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde: org.savy3.hadoop.hive.serde2.cobol.CobolSerde
hive>

hive> CREATE TABLE Cobol2Hive
> ROW FORMAT SERDE 'org.savy3.hadoop.hive.serde2.cobol.CobolSerde'
> LOCATION '/tmp/testdata/input_file.txt'
> TBLPROPERTIES ('cobol.layout'='01 WS-VAR. 05 WS-NAME PIC X(12). 05 WS-MARKS-LENGTH PIC 9(2). 05 WS-marks OCCURS 0 to 6 TIMES DEPENDING ON WS-MARKS-LENGTH. 10 WS-MARK PIC 999. 05 WS-NICKNAME PIC X(6)');
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde: org.savy3.hadoop.hive.serde2.cobol.CobolSerde

@manichinnari555: I recommend creating your own jar file from the code I provided. However for a quick test i have added jar file in location https://github.com/rbheemana/Cobol-to-Hive/tree/gh-pages/target
Serde expects the input file to be in binary format, could you please check file '/tmp/testdata/input_file.txt' is in binary format.

Thanks for the note.
Yes, we want to test with the sample data and the jar before we start customizing it.
Yes, it is binary data, however while creating the table, i am getting the following error. Do you have any idea about this error and how to fix it.
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde: org.savy3.hadoop.hive.serde2.cobol.CobolSerde

I will look into error, could you please send the sample file you are using.
Sent from Outlook on iOS
On Wed, Apr 13, 2016 at 10:42 AM -0700, "manichinnari555"<EMAIL_ADDRESS>wrote:
Thanks for the note.
Yes, we want to test with the sample data and the jar before we start customizing it.
Yes, it is binary data, however while creating the table, i am getting the following error. Do you have any idea about this error and how to fix it.
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde: org.savy3.hadoop.hive.serde2.cobol.CobolSerde
—
You are receiving this because you modified the open/close state.
Reply to this email directly or view it on GitHub

For now I am using the same file that you have provided in you blog. I want to see if i could use the file to create the table.
I think your jar works only the file layout that you have provided in your blog? is my understanding is correct?

No file needs to be in raw binary format. The example I provided in blog is for understand, actual binary files will be unreadable by human eye.
Jar works for any cobol layout.
To test u need to actually get the file from mainframe using the method I specified in blog. Just Creating a text file won't work as mainframe files are in EBCIDIC format not ASCII format.
Sent from Outlook on iOS
On Wed, Apr 13, 2016 at 10:56 AM -0700, "manichinnari555"<EMAIL_ADDRESS>wrote:
For now I am using the same file that you have provided in you blog. I want to see if i could use the file to create the table.
I think your jar works only the file layout that you have provided in your blog? is my understanding is correct?
—
You are receiving this because you modified the open/close state.
Reply to this email directly or view it on GitHub

Hello,
Got it, here is the binary file and the copybook layout that i am working with. Please let me if you need any more info.
bin_file.txt
CKNNEWTN.txt

Hello,
I have shared my file and copybook did you got a chance to look into that?
Sent from my iPhone

On Apr 13, 2016, at 2:08 PM, Ram Manohar Bheemana<EMAIL_ADDRESS>wrote:
No file needs to be in raw binary format. The example I provided in blog is for understand, actual binary files will be unreadable by human eye.
Jar works for any cobol layout.
To test u need to actually get the file from mainframe using the method I specified in blog. Just Creating a text file won't work as mainframe files are in EBCIDIC format not ASCII format.
Sent from Outlook on iOS
On Wed, Apr 13, 2016 at 10:56 AM -0700, "manichinnari555"<EMAIL_ADDRESS>wrote:
For now I am using the same file that you have provided in you blog. I want to see if i could use the file to create the table.
I think your jar works only the file layout that you have provided in your blog? is my understanding is correct?
—
You are receiving this because you modified the open/close state.
Reply to this email directly or view it on GitHub
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

Did you add jar to hive before you exceuted the create table?

Yes, I did added.
Add jar and the jar. Also copied to hive lib folder.
Sent from my iPhone

On Apr 14, 2016, at 3:48 PM, Ram Manohar Bheemana<EMAIL_ADDRESS>wrote:
Did you add jar to hive before you exceuted the create table?
—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub

I see things some things in query that needs to be fixed.

serde starts with com not org
Need to specify input format and output format
try placing the copybook in hdfs path since it is big and use cobol.layout.url
specify the fixed length property (if it is fixed length file)

It should something look like below:
ADD JAR CobolSerde.jar
CREATE EXTERNAL TABLE default.mani
ROW FORMAT SERDE 'com.savy3.hadoop.hive.serde2.cobol.CobolSerde'
INPUTFORMAT 'org.apache.hadoop.mapred.FixedLengthInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat'
LOCATION '/user/manichinnari/bin_file.txt'
TBLPROPERTIES ('cobol.layout.url'='/user/manichinnari/copybooks/CKNNEWTN.txt','fb.length'='450');

Hello,
Thanks for in detailed steps, I have tried these, however still facing same issue. Please let me know if we can take this offline and work on this.

Please connect me<EMAIL_ADDRESS>Sent from Outlook on iOS
On Fri, Apr 15, 2016 at 9:31 AM -0700, "manichinnari555"<EMAIL_ADDRESS>wrote:
Hello,
Thanks for in detailed steps, I have tried these, however still facing same issue. Please let me know if we can take this offline and work on this.
—
You are receiving this because you modified the open/close state.
Reply to this email directly or view it on GitHub

@manichinnari555: I cleaned up your comments as the issue is resolved by using  below link http://www.cloudera.com/documentation/enterprise/latest/topics/sg_sentry_service_config.html?scroll=concept_mvn_kvs_lr#concept_mvn_kvs_lr

Hello Ram,
Thanks, but I would like to let you know that, I am downloaded new SANBOX and executed previous steps that you have shared and it worked, but I am still not sure why I am getting the PIC clause issue, I am using the same copybook that I have share with you. If you have any information, please let me know, i will keep working on that. thanks.
Regards,

Hello Ram,
Things are much better now and lot of things are working, but "could you please add support of COMP and COMP4 variables"?
--
Regards,
Venkat

Hello Ram,
As discussed please find the attached copybook and the sample file.
bin_file.txt
CKNNEWTN.txt
Regards

Hello Ram,
Please find the updated COBOL
copy_book.txt
layout with all the fields init.
Regards,

Hello Ram,
Please find the describe output.
session.zip

bin_file_1.txt

Hello Ram,
Many Thanks for looking into the issue and everything seems working on. Really appreciated your help.
Regards,
Venkat

Hello Ram,
I have create the table1 using the cobol serde and table 2 with the same ddl but the storage format is different(parquet), later while trying to insert the data into table2 using the fololwing statement, I am getting the below error. Could you please have a look and guide me?
Is there any error in the SerDe?
Statement:
insert overwrite  table table2 PARTITION(dt='2016.06.13') select * from table1;
Error:
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row {"c5qtsys_detail_clnt_no":"2430","c5qtsys_detail_acct_no":"4510109900005012   ","c5qtsys_detail_log_date":20160406,"c5qtsys_detail_auth_log_time":151837,"c5qtsys_detail_maint_date":20151020,"c5qtsys_detail_maint_time":131009,"c5qtsys_detail_operator_id":"BLANMA","c5qtsys_detail_prev_frd_flag":"N","c5qtsys_detail_curr_frd_flag":"N","c5qtsys_detail_trans_amt":"+0000003.61","c5qtsys_detail_message_type":100,"c5qtsys_detail_reference_no":"    ","c5qtsys_detail_trans_date":2015289,"c5qtsys_detail_trans_id":"305289672360311 ","c5qtsys_detail_trans_apvl_cd":"010837","c5qtsys_detail_merchant_class":"8661","filler":"                                      "}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:545)
at org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:83)
... 17 more
Caused by: java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.hadoop.hive.serde2.io.ParquetHiveRecord
at org.apache.hadoop.hive.ql.io.parquet.write.ParquetRecordWriterWrapper.write(ParquetRecordWriterWrapper.java:124)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:753)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:838)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
at org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:164)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:535)

Hi Ram ,
I was trying to upload fixedformat cobol file into the hive table using the serde. But i got following error while query on the table.
select * from Cobol2Hive;
OK
Failed with exception java.io.IOException:java.io.IOException: Partial record(length = 62) found at the end of split.
Ascii File
Rajat       75SUNNY
Abhinandan  10B
Arpit       44JAIN
Ebcdic File
Ã<81><91><81>Â£@@@@@@@Ã·ÃµÃ¢Ã¤ÃÃÃ¨@%Ã<82><88><89><95><81><95><84><81><95>@@Ã±Ã°Ã@@@@@%Ã<99><97><89>Â£@@@@@@@Ã´Ã´ÃÃÃÃ@@
CobolCopybook
01 WS-VAR.
05 WS-NAME PIC X(12).
05 WS-EMPLOYEE-ID PIC 9(2).
05 WS-NICKNAME PIC X(6)

@rajatm91 First check the input file size. As it is a FB file, size of the binary file on hadoop should be multiple of 20 (based on copybook supplied). If it is not then the ftpied file is corrupted. Please try re-downloading.

Hello @rbheemana !
I really like your project, It looks very useful and I was thinking on use it but I have 2 doubts:

Is it possible to use the program to transform a EBCDIC file into a ASCII file in HDFS (instead of just read it as ASCII).
Is it possible to convert an ASCII file/table into an EBCDIC file/table?

Thank you very much in advance,
Best regards,
Pablo

I do not understand your questions completely, I believe you misunderstood the concept. But file in hdfs will be in EBCIDIC binary format. ASCII conversion happens only when we retrieve a column.
Thanks,
Ram

On Sep 27, 2016, at 04:37, randomusernamer<EMAIL_ADDRESS>wrote:
Hello @rbheemana !
I really like your project, It looks very useful and I was thinking on use it but I have 2 doubts:
Is it possible to use the program to transform a EBCDIC file into a ASCII file in HDFS (instead of just read it as ASCII).
Is it possible to convert an ASCII file/table into an EBCDIC file/table?
Thank you very much in advance,
Best regards,
Pablo
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Hi @rbheemana,
I understand the concept of the project. Indeed, I've used the program already to read a EBCDIC file via Hive.
My question is, given that the hardest part is solved (read an EBCDIC file and transform the information to ASCII using Hive), it is possible to use the class to write a file instead to show the information via Hive, Have you done something like this?
And the second question, right now the project reads an EBCDIC file and shows you a Hive table in ASCII format, Would it be possible to do the reverse process? i.e Would it be possible to read a Hive table to write an EBCDIC file in HDFS using this class?
Thank you for your time,
Best regards,
Pablo

Please see my response inline with your mail
Thanks,
Ram

On Sep 27, 2016, at 11:06, randomusernamer<EMAIL_ADDRESS>wrote:
Hi @rbheemana,
I understand the concept of the project. Indeed, I've used the program already to read a EBCDIC file via Hive.
My question is, given that the hardest part is solved (read an EBCDIC file and transform the information to ASCII using Hive), it is possible to use the class to write a file instead to show the information via Hive, Have you done something like this?

Simplest way to achieve this below, I have been suggesting same to everyone.
Step #1: Create a another hive table like the base table mentioning desired output file format. Ex: create table <new_table> like <old_table> stored as avro;
Step #2: Insert into <new_table> select * from <old_table>; This insert will convert the file format to desired format.

And the second question, right now the project reads an EBCDIC file and shows you a Hive table in ASCII format, Would it be possible to do the reverse process? i.e Would it be possible to read a Hive table to write an EBCDIC file in HDFS using this class?

I haven't seen any company interested in this. What would they do with EBCDIC format, put file on mainframe? There are lot of other ways that exists currently in mainframe world to convert ASCII to EBCDIC. If lot of people interested in this, it can be built easily.

Thank you for your time,
Best regards,
Pablo
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

Hi @rbheemana,
I was using the way you proposed, I was just asking If you have implemented the conversion to file to avoid read and interpret the copy again to create the table.
The interest of the ASCII to EBCDIC conversion is to replace some process of mainframe saving MIPS using the Hadoop cluster and then return the file to Host. We know many ways to convert ASCII to EBCDIC (in Host or zLinux) but we were looking for a solution that works for Hadoop to run the process in parallel.
Thanks,
Best regards,
Pablo

When I tried the solution you propose It throws an error:
Error: java.io.IOException: java.io.IOException: java.lang.NegativeArraySizeException
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:226)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.next(HadoopShimsSecure.java:136)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.moveToNext(MapTask.java:199)
at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.next(MapTask.java:185)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:52)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.io.IOException: java.lang.NegativeArraySizeException
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerChain.handleRecordReaderNextException(HiveIOExceptionHandlerChain.java:121)
at org.apache.hadoop.hive.io.HiveIOExceptionHandlerUtil.handleRecordReaderNextException(HiveIOExceptionHandlerUtil.java:77)
at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:355)
at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:105)
at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doNext(CombineHiveRecordReader.java:41)
at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.next(HiveContextAwareRecordReader.java:116)
at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.doNextWithExceptionHandler(HadoopShimsSecure.java:224)
... 11 more
Caused by: java.lang.NegativeArraySizeException
at com.savy3.mapreduce.MainframeVBRecordReader.nextKeyValue(MainframeVBRecordReader.java:175)
at com.savy3.mapred.MainframeVBRecordReader.next(MainframeVBRecordReader.java:63)
at com.savy3.mapred.MainframeVBRecordReader.next(MainframeVBRecordReader.java:1)
at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:350)
... 15 more
Do you have any idea what could it be happening?
Thank you very much,
Best regards,
Pablo

None so far.
Thanks,
Ram
This (https://itunes.apple.com/app/apple-store/id922793622?pt=814382&mt=8&ct=how_i_email)  is how I Email now

On Sep 28, 2016 at 04:47,  <randomusernamer (mailto:notifications@github.com)>  wrote:
Hi  @rbheemana (https://github.com/rbheemana),
I was using the way you proposed, I was just asking If you have implemented the conversion to file to avoid read and interpret the copy again to create the table.
The interest of the ASCII to EBCDIC conversion is to replace some process of mainframe saving MIPS using the Hadoop cluster and then return the file to Host. We know many ways to convert ASCII to EBCDIC (in Host or zLinux) but we were looking for a solution that works for Hadoop to run the process in parallel.
Thanks,
Best regards,
Pablo
—
You are receiving this because you were mentioned.
Reply to this email directly,   view it on GitHub (https://github.com/rbheemana/Cobol-to-Hive/issues/1#issuecomment-250108577), or   mute the thread (https://github.com/notifications/unsubscribe-auth/AMEsF9MpLWKbX7RCjYMKJtPpVqO62kNrks5quim8gaJpZM4GOG5X).

Looks like VB record reader has a bug. Please add a defect and provide me examples, will take a look when I get time.

How did you get the file into Hadoop?  I have found with the VB files you need to make sure that you transfer the files with the following settings:
binary
quote site RDW
quote site blksize=
If you do not do this, what happens is the limit is looking at the first split of the file and there is no issue.  The full query will return data across splits and without the block size it will not do the split correctly.  Normally when doing RDW the BDW (block delimiter) will come but when going from MF to Linux/windows the FTP process strips the BDW meaning it won't know when the block ends and the new one begins.
Remember that VB is varying character length but it is blocked so the blocks should be consistent.
Try that and if it does not work then there might be a bug.

It was a problem with the FTP, as you said the file was corrupt; after doing the FTP properly, I can see the data and materialize it in a file perfectly. All the selects works as expected. Thank you very much!!!
Best regards,
And again, thanks for your time

We have one ebcdic sample file.
It is stored in /user/hive/warehouse/ebcdic_test_file.txt
Cobol layout of the file is stored in /user/hive/Warehouse/CobolSerde.cob
We are running on Hue browser query editor.
We also tried in CLI.
But the same error is coming
We have added CobolSerde.jar.
Via
Add jar /home/cloudera/Desktop/CobolSerde.jar
It has been added successfully. Proof via LIST JARS.
Query
CREATE EXTERNAL TABLE cobol2Hve
ROW FORMAT SERDE 'com.savy3.hadoop.hive.serde2.cobol.CobolSerDe'
STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.FixedLengthInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat'
LOCATION '/user/hive/warehouse/ebcdic_test_file.txt'
TBLPROPERTIES ('cobol.layout.url'='/user/hive/warehouse/CobolSerDe.cob','fb.length'='159');
Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. Cannot validate serde: com.savy3.hadoop.hive.serde2.cobol.CobolSerDe

Why is the error coming?
What is fb.length?

Hi Bheemana, This is a very good tool for many of us. You guys are great.
I just have few questions in using this tool.

Both the input file and its corresponding layout / copybook file should be in EBCDIC format or only the input file should be in EBCDIC and the layout can be in .txt format ?
And the input file along with its layout file should be available in HDFS or the CobolSerde can fetch them from Unix file system as well. I assume both should be in HDFS.

Please confirm

Input file should be in EBCDIC format, copy book in text file. Both files needs to be hdfs.
Thanks,
Ram

On Apr 13, 2017 at 3:00 AM,  <jkckiran (mailto:notifications@github.com)>  wrote:
Hi Bheemana, This is a very good tool for many of us. You guys are great.
I just have few questions in using this tool.
Both the input file and its corresponding layout / copybook file should be in EBCDIC format or only the input file should be in EBCDIC and the layout can be in .txt format ?
And the input file along with its layout file should be available in HDFS or the CobolSerde can fetch them from Unix file system as well. I assume both should be in HDFS.
Please confirm
—
You are receiving this because you were mentioned.
Reply to this email directly,   view it on GitHub (https://github.com/rbheemana/Cobol-to-Hive/issues/1#issuecomment-293809183), or   mute the thread (https://github.com/notifications/unsubscribe-auth/AMEsF_aRUws0jYLJloUyhxlu1I0m-WGeks5rvcgLgaJpZM4GOG5X).

I have this error when i try to execute this : "Unsupported major.minor version 52.0"
Any help ?
John

Hi @rbheema,
Thank you for reply but I don’t think these comments answer to my issue.
My all files are stored on HDFS. The error seems to have link to a version issue but I don’t know which version.
May be java version between jar and server ?
John
Provenance : Courrierhttps://go.microsoft.com/fwlink/?LinkId=550986 pour Windows 10
De<EMAIL_ADDRESS>Envoyé le :mercredi 31 mai 2017 14:53
À<EMAIL_ADDRESS>Cc<EMAIL_ADDRESS><EMAIL_ADDRESS>Objet :Re: [rbheemana/Cobol-to-Hive] (post comments) Intoduction to Cobol to Hive Serde blogs (#1)
Hello,
Please find the inline comments.
Sent from my iPhone

On Apr 13, 2017, at 3:00 AM, jkckiran<EMAIL_ADDRESS>wrote:
Hi Bheemana, This is a very good tool for many of us. You guys are great.
I just have few questions in using this tool.
Both the input file and its corresponding layout / copybook file should be in EBCDIC format or only the input file should be in EBCDIC and the layout can be in .txt format ?
Ven: only input file should be EBCDIC and the layout in txt.
And the input file along with its layout file should be available in HDFS or the CobolSerde can fetch them from Unix file system as well. I assume both should be in HDFS.
Yes, both should be in hdfs.
Please confirm
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://github.com/rbheemana/Cobol-to-Hive/issues/1#issuecomment-305177110, or mute the threadhttps://github.com/notifications/unsubscribe-auth/Aa-vP45IqehOQA6QCTh0k3KeYmF-Pp6nks5r_WKngaJpZM4GOG5X.

I finally succesed yesterday to create my table with cobol serde by execute it in the hive shell instead of HUE.
But when I do a "select" on my created table, all values are null
NULL    NULL    NULL    ????????????&????????   NULL    ??????? ????????????    ?????????       NULL    NULL    NULL    NULL    NULL    NULL    NULL    ?       ?      ??       ??      ???     NULL    NULL    ?       NULL    NULL    ??      ???     ?       NULL    NULL    ??????????      ?       ??      NULL    NULL    NULL    NULL   NULL     ?       ??????????      NULL    ?       NULL    NULL    ??????????????????????  NULL    NULL    ??
NULL    NULL    NULL    ????????????&???????    NULL    ??????? ????????????    ?????????       NULL    NULL    NULL    NULL    NULL    NULL    NULL    ?       ?      ??       ??      ???     NULL    NULL    ?       NULL    NULL    ??      ???     ?       NULL    NULL    ??????????      ?       ??      NULL    NULL    NULL    NULL   NULL     ?       ??????????      NULL    ?       NULL    NULL    ??????????????????????  NULL    NULL    ??
Anyone get this happenned ?
John

Hi Ram
i have the jar in my lib folder and ran the following command from the lib path with the actual values  but i am getting the following error and not sure why . Can you please let me know if you know what the reason might be ? I would really appreciate it. Thanks!
hadoop jar CobolSerde.jar com/savy3/util/FTPtoHDFS mainframe:port username password mainframeinputfilename hadooppathforoutput
Connected to osa-test.nfcu.net on 21
reply for LITERAL200 SITE command was accepted
FTP Start Time : 2017-06-01 17:30:39.124
Exception in thread "main" java.lang.NullPointerException
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:85)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:59)
at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:119)
at com.savy3.util.FTPtoHDFS.main(FTPtoHDFS.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

@johnplt Please follow the instructions in this link to compile the code corectly https://github.com/rbheemana/Cobol-to-Hive/issues/20
The issue you mentioned arises when the Java version on which you compiled is different from Java version on which you executed.
Please try compiling again and check if the NULL issue still persists.

@sanch002 could you run manual FTP commands from the edge node, to verify you can establish FTP connection to FTP server. I believe Firewall port are not open to make an FTP connection from your cluster to FTP server.

Thanks for the reply @rbheemana. I failed to create the jar file. It seems that it does not success to find some packages, Example : package org.apache.hadoop.hive.serde2.objectinspector does not exist.
May be it's the definition of the hadoop home and hive home (set %classpath%=%HADOOP_HOME%;%HIVE_HOME%\lib;)
I am on Unix and I define the classpath like that : classpath=/usr/bin/hadoop:/usr/bin/hive
That seems to be wrong. Anyone nows how to retrieve the hadoop and hive home ?

@johnplt Please check with your hadoop adminstrator, different distributions have different locations. Also it can be custom defined. So there is no single answer for it. Best bet is to check with person who installed your cluster.

Hi Ram,
What special care need to be taken to load variable length file?
have created sample file given in problem description and  FTP'ed using option 6 in mainframe in binary format. created cobol copybook is also as given. But, still getting Partial record (length = ***) found at the end of split error.
ADD JAR /CobolSerde.jar;
CREATE EXTERNAL TABLE test
ROW FORMAT SERDE 'com.savy3.hadoop.hive.serde3.cobol.CobolSerDe'
STORED AS
INPUTFORMAT 'com.savy3.mapred.MainframeVBInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat'
LOCATION '/user/test'
TBLPROPERTIES ('cobol.layout.url'='/user/copybook/cobol.cp')
copybook is --
01 WS-VAR.
05 WS-NAME PIC X(12).
05 WS-MARKS-LENGTH PIC 9(2).
05 WS-marks OCCURS 0 to 25 TIMES DEPENDING ON WS-MARKS-LENGTH.
10 WS-MARK PIC 999.
05 WS-NICKNAME PIC X(6).

Please download with RDW byte to determine length of each record

On Jul 27, 2017 at 11:47 AM,  <shankardh (mailto:notifications@github.com)>  wrote:
Hi Ram,
What special care need to be taken to load variable length file?
have created sample file given in problem description and FTP'ed using option 6 in mainframe in binary format. created cobol copybook is also as given. But, still getting Partial record (length = ***) found at the end of split error.
ADD JAR /CobolSerde.jar;
CREATE EXTERNAL TABLE test
ROW FORMAT SERDE 'com.savy3.hadoop.hive.serde3.cobol.CobolSerDe'
STORED AS
INPUTFORMAT 'com.savy3.mapred.MainframeVBInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat'
LOCATION '/user/test'
TBLPROPERTIES ('cobol.layout.url'='/user/copybook/cobol.cp')
copybook is --
01 WS-VAR.
05 WS-NAME PIC X(12).
05 WS-MARKS-LENGTH PIC 9(2).
05 WS-marks OCCURS 0 to 25 TIMES DEPENDING ON WS-MARKS-LENGTH.
10 WS-MARK PIC 999.
05 WS-NICKNAME PIC X(6).
—
You are receiving this because you were mentioned.
Reply to this email directly,   view it on GitHub (https://github.com/rbheemana/Cobol-to-Hive/issues/1#issuecomment-318402882), or   mute the thread (https://github.com/notifications/unsubscribe-auth/AMEsF-9kkb49n0O6P2lnvKWuuMqERFmYks5sSLEdgaJpZM4GOG5X).

Hi Bheemana
Thank you for creating a very useful SERDE. I have variable blocked file of large size and tons of columns. Just imported the project and performed mvn install to obtain the jar. Before proceeding I have these questions:

The built jar is named Cobol-to-Hive-1.1.0.jar but your instruction says CobolSerde.jar. Did I miss something? But I see all the required classes in Cobol-to-Hive-1.1.0.jar.

Should I transfer the VB file "as is" to HDFS or have it unblocked? You mentioned RDW but the file contains a BDW at the beginning of each block with multiple RDW. Is there any pre-processing required?

COBOL can throw a lot of curve balls like RENAMES, CONDITION-NAMEs, PIC with SIGN, zoned decimal, packed decimal with sign etc.It would be good to know any  limitations in features and size. Are there any I need to be aware of?

Does it handle multipe ODO (Occurs Depending Upon)  in each record?

Is there a file size / max column limitations? I am working on files that are 30GB and more. Some of them have 4K columns.

I would appreciate a response at your earliest. Thanks

Error: Error while compiling statement: FAILED: SemanticException 1:14 Input format must implement InputFormat.  How to resolve this issue. HDP 3 environment.

@any2xml

The built jar is named Cobol-to-Hive-1.1.0.jar but your instruction says CobolSerde.jar. Did I miss something? But I see all the required classes in Cobol-to-Hive-1.1.0.jar.
1.1.0 is the updated version

Should I transfer the VB file "as is" to HDFS or have it unblocked? You mentioned RDW but the file contains a BDW at the beginning of each block with multiple RDW. Is there any pre-processing required?
I believe BDW should also work, I have not tried. Let me know if you face any issues.

COBOL can throw a lot of curve balls like RENAMES, CONDITION-NAMEs, PIC with SIGN, zoned decimal, packed decimal with sign etc.It would be good to know any  limitations in features and size. Are there any I need to be aware of?
It solves most of them, but I haven't handled for RENAMES and CONDITION-NAMEs. Create an issue for desired functionality, i will try to code in my free time.

Does it handle multipe ODO (Occurs Depending Upon)  in each record?
Yes it does.

Is there a file size / max column limitations? I am working on files that are 30GB and more. Some of them have 4K columns.
No limitations.

@rajeshmese Please let me know the steps how you tried to compile

Hi I tried to create a table as below, but I get an error , can anyone please help.
Some notices:
Jar file (CobolSerde.jar) is in hdfs
datafile.txt is a text file which is in hdfs aswell
COPYBOOK.TXT is the copybook which is a text file residing in hdfs as well.
ADD JAR /user/cloudera/copybooks/CobolSerde.jar;
CREATE EXTERNAL TABLE cobol2hive
ROW FORMAT SERDE 'com.savy3.cobolserde.CobolSerde'
STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.FixedLenghtInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat'
LOCATION '/user/hive/warehouse/datafile.txt'
TBLPROPERTIES ('cobol.layout.url'='/user/cloudera/copybooks/COPYBOOK.TXT','fb.length'='130');
--Running this command:
ADD JAR /user/cloudera/copybooks/CobolSerde.jar;
delivers the following error:
AnalysisException: Syntax error in line 1:undefined: ADD JAR /user/cloudera/copybooks/CobolSerde.jar ^ Encountered: ADD Expected: ALTER, COMPUTE, CREATE, DELETE, DESCRIBE, DROP, EXPLAIN, GRANT, INSERT, INVALIDATE, LOAD, REFRESH, REVOKE, SELECT, SET, SHOW, TRUNCATE, UPDATE, UPSERT, USE, VALUES, WITH CAUSED BY: Exception: Syntax error
--Running the whole code altogether returns this Error:
AnalysisException: Syntax error in line 3:undefined: ROW FORMAT SERDE 'com.savy3.cobolserde.CobolSerde' ^ Encountered: IDENTIFIER Expected: DELIMITED CAUSED BY: Exception: Syntax error

Hello All,
I need a favour. I am not able to recreate the jar.
Can someone please share jar or the location with me ASAP, really appreciated your help with this.
Regards,
Venkat

On May 8, 2019, at 4:57 PM, thenervousgeek<EMAIL_ADDRESS>wrote:

Hi I tried to create a table as below, but I get an error , can anyone please help.
Some notices:
Jar file (CobolSerde.jar) is in hdfs
datafile.txt is a text file which is in hdfs aswell
COPYBOOK.TXT is the copybook which is a text file residing in hdfs as well.
ADD JAR /user/cloudera/copybooks/CobolSerde.jar;
CREATE EXTERNAL TABLE cobol2hive
ROW FORMAT SERDE 'com.savy3.cobolserde.CobolSerde'
STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.FixedLenghtInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat'
LOCATION '/user/hive/warehouse/datafile.txt'
TBLPROPERTIES ('cobol.layout.url'='/user/cloudera/copybooks/COPYBOOK.TXT','fb.length'='130');
--Running this command:
ADD JAR /user/cloudera/copybooks/CobolSerde.jar;
delivers the following error:
AnalysisException: Syntax error in line 1:undefined: ADD JAR /user/cloudera/copybooks/CobolSerde.jar ^ Encountered: ADD Expected: ALTER, COMPUTE, CREATE, DELETE, DESCRIBE, DROP, EXPLAIN, GRANT, INSERT, INVALIDATE, LOAD, REFRESH, REVOKE, SELECT, SET, SHOW, TRUNCATE, UPDATE, UPSERT, USE, VALUES, WITH CAUSED BY: Exception: Syntax error
--Running the whole code altogether returns this Error:
AnalysisException: Syntax error in line 3:undefined: ROW FORMAT SERDE 'com.savy3.cobolserde.CobolSerde' ^ Encountered: IDENTIFIER Expected: DELIMITED CAUSED BY: Exception: Syntax error
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

From below error, looks like you do not have ADD permissions.
On Wed, Dec 11, 2019 at 11:47 AM manichinnari555<EMAIL_ADDRESS>wrote:

Hello All,
I need a favour. I am not able to recreate the jar.
Can someone please share jar or the location with me ASAP, really
appreciated your help with this.
Regards,
Venkat

On May 8, 2019, at 4:57 PM, thenervousgeek<EMAIL_ADDRESS>wrote:

Hi I tried to create a table as below, but I get an error , can anyone
please help.
Some notices:
Jar file (CobolSerde.jar) is in hdfs
datafile.txt is a text file which is in hdfs aswell
COPYBOOK.TXT is the copybook which is a text file residing in hdfs as
well.
ADD JAR /user/cloudera/copybooks/CobolSerde.jar;
CREATE EXTERNAL TABLE cobol2hive
ROW FORMAT SERDE 'com.savy3.cobolserde.CobolSerde'
STORED AS INPUTFORMAT 'org.apache.hadoop.mapred.FixedLenghtInputFormat'
OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.IgnoreKeyTextOutputFormat'
LOCATION '/user/hive/warehouse/datafile.txt'
TBLPROPERTIES
('cobol.layout.url'='/user/cloudera/copybooks/COPYBOOK.TXT','fb.length'='130');
--Running this command:
ADD JAR /user/cloudera/copybooks/CobolSerde.jar;
delivers the following error:
AnalysisException: Syntax error in line 1:undefined: ADD JAR
/user/cloudera/copybooks/CobolSerde.jar ^ Encountered: ADD Expected: ALTER,
COMPUTE, CREATE, DELETE, DESCRIBE, DROP, EXPLAIN, GRANT, INSERT,
INVALIDATE, LOAD, REFRESH, REVOKE, SELECT, SET, SHOW, TRUNCATE, UPDATE,
UPSERT, USE, VALUES, WITH CAUSED BY: Exception: Syntax error
--Running the whole code altogether returns this Error:
AnalysisException: Syntax error in line 3:undefined: ROW FORMAT SERDE
'com.savy3.cobolserde.CobolSerde' ^ Encountered: IDENTIFIER Expected:
DELIMITED CAUSED BY: Exception: Syntax error
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or mute the thread.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/rbheemana/Cobol-to-Hive/issues/1?email_source=notifications&email_token=ADASYF25O42CRYWTXW623X3QYEKS5A5CNFSM4BRYNZL2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEGTZXNY#issuecomment-564632503,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADASYFZOEJENRTH3DWYE46TQYEKS5ANCNFSM4BRYNZLQ
.

Hello Ram,
Can you please help with the jar?
Also, I see in the git some part of the code is commented out. Can you please help me?
Regards,
Venkat

On Dec 11, 2019, at 11:47 AM, chinnari venkati brahmam<EMAIL_ADDRESS>wrote:


feat: exporter collector TLS option

Resolves this: https://github.com/open-telemetry/opentelemetry-js/issues/1062

Codecov Report

Merging #1063 into master will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master    #1063   +/-   ##
=======================================
  Coverage   95.12%   95.12%
=======================================
  Files         213      213
  Lines        8950     8951    +1
  Branches      806      806
=======================================
+ Hits         8514     8515    +1
  Misses        436      436

Impacted Files
Coverage Δ

...emetry-exporter-collector/src/CollectorExporter.ts
90.47% <100.00%> (+0.23%)
:arrow_up:

Certs exist only for testing purposes yes? If so they should be in the test directory.

Certs exist only for testing purposes yes? If so they should be in the test directory.

Good point, fixed. Let me know if you have any other comments.

@obecny would like to get your eyes on this.

Codecov Report

:exclamation: No coverage uploaded for pull request base (master@637f982). Click here to learn what that means.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##             master    #1063   +/-   ##
=========================================
  Coverage          ?   73.40%
=========================================
  Files             ?       71
  Lines             ?     1534
  Branches          ?      264
=========================================
  Hits              ?     1126
  Misses            ?      408
  Partials          ?        0

Impacted Files
Coverage Δ

...ry-exporter-collector/src/CollectorExporterBase.ts
90.00% <100.00%> (ø)

.../opentelemetry-exporter-collector/src/transform.ts
95.71% <100.00%> (ø)

...ntelemetry-core/src/platform/BaseAbstractPlugin.ts
60.00% <0.00%> (ø)

...ackages/opentelemetry-api/src/metrics/NoopMeter.ts
73.17% <0.00%> (ø)

...ackages/opentelemetry-api/src/trace/trace_flags.ts
100.00% <0.00%> (ø)

...y-core/src/context/propagation/HttpTraceContext.ts
100.00% <0.00%> (ø)

...ages/opentelemetry-core/src/internal/validators.ts
100.00% <0.00%> (ø)

...es/opentelemetry-context-zone-peer-dep/src/util.ts
100.00% <0.00%> (ø)

...st/trace/fixtures/test-package/foo/bar/internal.js
0.00% <0.00%> (ø)

...y-exporter-collector/test/browser/index-webpack.ts
100.00% <0.00%> (ø)

... and 63 more

@obecny Thanks a lot for the detailed feedback! Fixed all the comments and checked as you suggested with web example - seems to work fine. Let me know if you have any other comments.


[Mobile wallet] Send MWC via slatepack feature, previously provided slatepack messages are getting displayed.

Description:
In Mobile wallet, Send MWC via slatepack feature, previously provided slatepack messages are getting displayed.
Steps to reproduce

Initiate Send MWC via slatepack feature from Mobile wallet.
Finalize the transaction.
Initiate another Send MWC transaction via slatepack from Mobile wallet.
Observed previously provided slatepack messages are getting displayed.

Please refer to the below screenshot for more details.

Expected Result:
Previous transactions slatepack messages should be pre-filled in the next transactions.

Verified with the latest build, hence closing this ticket.


Security vulnerability affecting all versions

A security vulnerability in postcss-loader has been identified by synk.io (see https://security.snyk.io/vuln/SNYK-JS-POSTLOADER-2403737).

We postcss-loader, you provide link on post-loader, it is diffident package

Can you show a message?

Can you write npm ls postcss-loader and npm ls post-loader?

Could it be possible, that it was incorrectly assigned to this package (npm and JFrog XRay)?
Here is the correct link to Snyk:
https://security.snyk.io/vuln/SNYK-JS-POSTLOADER-2403737

You should update postcss-loader to https://github.com/webpack-contrib/postcss-loader/blob/master/package.json#L3, maybe even open an issue in angular cli

Yes we also need to do this. But version 6.2.1 is also blocked by jfrog and npm audit. I will open an issue at angular cli to address this incorrect vulnerability.


how to use FoldingFactorizationBasedRecommender

hey,
I tried many times to use the recommend er mentioned above with SvdPlusPlusrecommender implemented by mahout, but everytime i fail to get recommendation since i do not know how to pass parameters of the recommender and HOW to fold new users ( like totally new users, or an old user with new rated items).
could you share us a use example ?

one of tries i create a persist factorization by svd++ recommender, i passed it to foldingfactorizationrecommender, I GOT an exception telling that the matrix is not singular.
any help please

DataModel model= new FileDataModel(new File("src/u.data"));
//SVDPlusPlusFactorizer factorizer=new SVDPlusPlusFactorizer(model, 10, 10);
	  ALSWRFactorizer factorizer= new ALSWRFactorizer(model, 10,0.05, 10);

      Factorization factorization= factorizer.factorize();

      EuclideanDistanceSimilarity eds = new EuclideanDistanceSimilarity(model);
      CandidateItemsStrategy candidateItemsStrategy= new AllSimilarItemsCandidateItemsStrategy(eds);
      PersistenceStrategy persistence =
    		  new FilePersistenceStrategy(new File("src/test"));
      persistence.maybePersist(factorization);
      FoldingFactorizationBasedRecommender ffbr =new
    		  FoldingFactorizationBasedRecommender(model,candidateItemsStrategy,persistence);

svd++ doesn't work, als seems good, any additional explanation about fold in new users ?

U have to pass config file on startup:
mvn -Dkornakapi.conf=/.../kornakapi.conf clean tomcat:run
About folding in new user, u can only make recommendations for users and items that already have a feature vector estimated. In case there is a new user you know nothing about(no associated item) then you cant apply the model to make predictions. However if you know at least one item about the user then you can try to estimate the users feature vector by that items feature vector.

thank you very much for your reply is so useful, but I am not launching a tomcat server but using the recommender only locally for tests.  I created the code above there is not more exception of: singular matrix" related to the problem of the inability to compute the inverse matrix.

is it possible to update the model after a user rates new items ?

Yes, that possible, the proper way would be to retrain the hole model,
however you could also just recalculate the user feature vector.

Sorry i did not ment to close


Fix/is reactive

fix issue
#74
code

Wrap ReactiveFlags's members with Symbol
Make ReactiveFlags a const value

mini-vue 只实现 happy path 逻辑哈

mini-vue 只实现 happy path 逻辑哈


Possibility of managing users in server side?

In this documentation we know how to manage users using JavaScript API:
https://github.com/P2Pvalue/swellrt/wiki/Working-with-users
But is it possible to do the same things in server side? Thank you.

Sorry, at this moment the SwellRT server doesn't provide an administrative
console. That is a future feature.
2017-11-29 2:57 GMT+01:00 ngwaitat<EMAIL_ADDRESS>
In this documentation we know how to manage users using JavaScript API:
https://github.com/P2Pvalue/swellrt/wiki/Working-with-users
But is it possible to do the same things in server side? Thank you.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/P2Pvalue/swellrt/issues/226, or mute the thread
https://github.com/notifications/unsubscribe-auth/AEP6D_MbMEB0nTzmMfbGULOPvVb2rzuOks5s7LocgaJpZM4QuTfp
.

I see, thank you very much.


⚠ api.piste.gouv.fr healthcheck has degraded performance

In 882be7c, api.piste.gouv.fr healthcheck (https://api.piste.gouv.fr/cassation/judilibre/v1.0/healthcheck) experienced degraded performance:

HTTP code: 200
Response time: 4835 ms

Resolved: api.piste.gouv.fr healthcheck performance has improved in 0a22f89.


Swagger / Yard Docs

Has anyone been able to integrate/document the paths here with Swagger or is there a reason not to? (Security/Best Practice/etc)

Could be api blueprint ? Because I can do that

No discussion since March, closing for now. Feel free to reopen if this interests you!

How use grape-swagger-rails gem to document authentication with devise auth token?????

https://www.javavines.com/2021/12/swagger-file-devise-token-auth.html


disabled prop is not working on IOS

disabled={true} is not working on IOS, is it tends to work only for IOS and not for android ?

On android, It is not working properly.
When I navigate to a screen that uses phone-number-input among other input fields, all of the fields disappear a huge input field with a place holder "Enter country name" appears instead where it is not possible to type or edit in it.
When disabled={false} everything works fine.
Any ideas? help?

same issue on android.

I make a PR #26 to resolve it. @garganurag893 could it be merged soon?

I still have this issue @garganurag893, please can you merge @nachozullo's PR

@nachozullo Your PR fix didn't work for me


Fixed expansion edge case failure

When argument overlaps with the trigger, backoff from expansion to the
original.  Gives a debug statement instead of a error.
@MihaiSurdeanu fyi
if travis is happy, i think closes #364
??

I can do this too -- the one sentence I saw was a really broken syntax
sentence... but i can poke around some more
On Thu, Aug 16, 2018 at 10:21 AM Keith Alcock<EMAIL_ADDRESS>wrote:

I should probably figure out which sentences were causing problems in
those documents and add one as a test case. We don't usually run into them
in the present test cases or the 52 documents.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/clulab/eidos/pull/412#issuecomment-413621293, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFIniVmJ3hmIvSess2XJSceucC9tcUbBks5uRaoVgaJpZM4WAQgX
.

hold up -- please don't merge quite yet...

This is a cleaner fix -- but, i don't know which sentence I should make a test for.... there was a lack of specificity in some triggers (one issue on the trigger side, another with a rule), so...? Should I just grab any of the previously failing sentences?

I don't know any better.  I was just going to verify that the code has been visited once and doesn't throw a new exception or otherwise blow up.

So, as we discussed I'm starting another run across the 10 or so problem documents, the collection of 52, and then the 10K to check for issues.

woohoo! vaya con dios!
On Fri, Aug 17, 2018 at 10:44 AM Keith Alcock<EMAIL_ADDRESS>wrote:

So, as we discussed I'm starting another run across the 10 or so problem
documents, the collection of 52, and then the 10K to check for issues.
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/clulab/eidos/pull/412#issuecomment-413939547, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AFIniS8ZwMMz1jI7OFsThLENCW9Dgq-dks5uRwEVgaJpZM4WAQgX
.

It looked good on the first half last night, but waiting until morning I find error reports for these files:
2009_Document_a-bq992e.txt
2017_Document_a-i8219e.txt
2005_Book_a-am390e.txt
2013_Meeting_a-bk305e.txt
1964_Document_a-bm438e.txt
1987_Document_a-bm745e.txt
1978_Document_a-bm334e.txt
2008_Meeting_a-k2554e.txt
2017_Reports_a-bu002e.txt
Don't hesitate to request more details or help if it would be useful or if this can't be reproduced.  And ignore this until Monday, of course.

ok -- so, how about this.... i removed the exception bc Odin seems to gracefully handle the situation all by itself, so who am I to complain, right?
Aside from that, this journey did illuminate some issues with missing triggers so... yay?
@kwalcock please check for me and see, since i removed the exception, not it just gives a debug message and moves on, is this ok?

The latest version made it through 10K+ documents without complaining and not causing other obvious problems due to its not complaining.  So, I'll merge it momentarily.  Thanks for working on it.


[WIP] - Fix link redirection

This is still a work in progress because every time that i scratch the surface of links, i found new edge cases that need to be fixed/covered.
As i said during the sprint,  component doesn't work well with links outside the router routes. We also don't need the router with external links.
I fixed LinkView checking if the url has the same domain as the window.location one.
Probably in the future this check should be overriden by a simple check on how the link string is made (when we implement internal links into Link objects).
btw right now it kinda works.
The other thing that i wanted to fix, is the link generated into Navigation component.
In Plone, if the user has the permission to edit that object, the link in Navigation is the url of the object itself, otherwise, if the user can't edit it, the url is remoteUrl.
I created a new wrapper component for manage links: this check if the link is internal or external, and creates a simple  or a router's .
It's a proof of concept because i don't know how you want to handle links into Volto.
There are some problems:

The first (and most important) is that right now in "isExternalLink" function i'm checking the link's domain with the one from window.location, but in SSR we don't have "window" so i cheated using the apiPath. This could probably fixed in the near future when we develop link's widget for handling internal and external links.
Links should have different behaviors if the current user can edit it or not. Do we want to emulate all the Plone features (for example in Navigation show absolute_url or remoteUrl, in contents view, always show the absolute_url and then let the view make the redirect) or we decide that in Volto they should work in a different way?

@cekk this can be closed, right?

Closing since this is quite old and appears to overlap with functionality that was already implemented. Let us know if there's anything here that is still important.


Update Elasticsearch to 5.5.1

fix example file path and some log information
modify yaml file in order to run 5.5.1

@k8s-ci-robot I signed it!

@pires do you mind taking a look here, since this is using your images ?

Please, upgrade to latest 5.6.2 and then I'll take a look.

Hi @pires
I make some change, please check it.
Thank you~~~

/lgtm


Stub out credential creation process

Similar to how in #7, the Spec function is stubbed out, do so for the remaining parts of the credentials creation pipeline, as well as validation of credentials.
Stubbing out means the types are there but the implementation isn't. It's a very useful exercise because it makes the interdependencies and overall architecture apparent

blocked on final protocol spec


added test files to folders to get them to push up

test files created, looks like the folders are there now

Looks good! Thanks


[Personalizer] Release prep

Fixing the builds to release the personalizer package

/azp run js - personalizer - ci

/azp run prepare-pipelines

/azp run js - personalizer

/azp run js - personalizer - ci

/azp run js - personalizer - ci

/check-enforcer evaluate

/check-enforcer evaluate

API change check
APIView has identified API level changes in this PR and created following API reviews.
azure-rest-ai-personalizer


[Question] NVIDIA GeForce RTX 4080 GPU is not yet supported in the Docker

:question: Question

Hi, I build and run the docker, everything looks right except a warning that stated Detected NVIDIA NVIDIA GeForce RTX 4080 GPU, which is not yet supported in this version of the container
ERROR: No supported GPU(s) detected to run this container

Is there any solution to this problem ?

You could try to bump the base image in the docker image to a newer PyTorch version (make sure to keep the PyTorch Version below 2.0 since it is not yet supported in nnDetection).


skia performance monitoring misattributing regressions

Twice today we've seen regression in performance that after investigation weren't caused by the commit where the regression appeared in skia's performance graphs.  They were attributed to framework changes but appear to be in engine roll commits.
Examples:
https://github.com/flutter/flutter/issues/68316
https://github.com/flutter/flutter/issues/68317
cc @kf6gpe @jonahwilliams @liyuqian

Further investigation gave plausible reasons for the failures we've seen.  I don't think this is actually the case.


Data scripts to json

Summary of Changes

Edits to the get_emissions.py file that just:

Reads in the raw data
Feeds raw into the helper functions
prints the output of the helper as a json object

Created the get_generation.py file that does the exact same thing as described above.

Edits to utils.py that:

only keeps the columns we need for respective data
sets the multi index of (state, year)
removes unnecessary comments in code

Expected Output
The json object output for final should look like this:
{'metric_label':
     {"(`state', year)":actual_value,
      for all 50 states for all years....},
{'new_metric_label': etc...}

Ready for your review whenever @derekeder

looks like it failed to build because it thinks the data/decarbonize_us_analysis is a submodule. I removed the folder since we pulled out what we need from it.

I removed old references to the allOccupationsJson that we're no longer using to get the builds to pass, but it looks like we'll need to get the emissions.json data in shape before making any more progress on it.
@swatland610 once that data is prepped, let me know and I'll work to get the site building with the new data

@derekeder Alright, I've added a helper in the utils.py file.
I've also adjusted both the get_data files to remove some redundancies. Let me know if the output is not working for you! I got it working this morning.

making progress!

charts!

yay - its building now. I'm going to merge this in now so we have something to share with the group. I'll open an issue with a few requests for the data.
closes #2
closes #4
closes #3


About the usage of function "maybe_no_sync" in gsam

Hi there, thanks for your work. I notice that you use the "no_sync" function in gsam.py(https://github.com/juntang-zhuang/GSAM/blob/main/gsam/gsam.py#L128). I wonder why should it be used. If we use 4 GPUs and 256 data per GPU, the gradient after sync is the correct gradient calculated based on 256*4 data, and the gradients would be calculated based on 256 if we set no_sync. Is this lead to a slight error.

The perturbation is per-worker (same as SAM), so each worker needs to have a different grad. After the per-worker perturbation, and get the per-worker new grad, then another sync_grad is called https://github.com/juntang-zhuang/GSAM/blob/704942daa5654a2dd08118d7a32a1dd8afdf149e/gsam/gsam.py#L180 to make sure all workers have the same update

The perturbation is per-worker (same as SAM), so each worker needs to have a different grad. After the per-worker perturbation, and get the per-worker new grad, then another sync_grad is called
https://github.com/juntang-zhuang/GSAM/blob/704942daa5654a2dd08118d7a32a1dd8afdf149e/gsam/gsam.py#L180
to make sure all workers have the same update

Thanks for your reply. Have you tested the SAM without per-worker, i.e., each worker uses the same grad and calculates the same perturbation?

I think the original SAM paper tested this, and call samples_per_worker as m in their paper. Turns out a smaller m improves the results a lot.

I think the original SAM paper tested this, and call samples_per_worker as m in their paper. Turns out a smaller m improves the results a lot.
Thank you so much. This per-worker setting has confused me for a while. Thank you for releasing the code. Maybe you can close this issue.

@juntang-zhuang

I think the original SAM paper tested this, and call samples_per_worker as m in their paper.

I did not see the related experiment in their paper, could you help to point out?
by the way, samples_per_worker means the batch size per gpu?

See sec 4.1 in the original SAM paper, "m-sharpness". sampler_per_worker generally means batchsize per GPU, if your model can be fitted into one GPU.
In some cases one model needs to be sharded across multiple GPUs, in that case one replica means one full shard of GPUs such that they together hold one complete model.


docs: fix out-of-dated property name in Config/Modules.php

Description
$activeExplorers was renamed in https://github.com/codeigniter4/CodeIgniter4/commit/3626206ca7784bd6d63e0daecbf9484ac302319e.
Checklist:

[x] Securely signed commits
[ ] Component(s) with PHPdocs
[ ] Unit testing, with >80% coverage
[x] User guide updated
[ ] Conforms to style guide

Thank you @kenjis


Fix typo in en.yaml

I noticed a small typo: avaliable <> available

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.


Error while Integrating with slack

Rasa Core version: = 0.11.5
Python version:  2.7
Operating system (windows, osx, ...): unix
Issue:  When i am giving request URL it is giving error message. Your request URL gave us a 500 error. Update your URL to receive a new request and challenge value.
Please help me to fix this issues.
Content of domain file (if used & relevant):

@mk424 please tell me how to fix error above


Compatiblity with bootstrap 4

Hi, in Bootstrap 4 (alpha) the class 'open' is replaced by 'show'. To make it work in version 4,  I replaced it in your code,and it works fine.
Thanks for the great component.

Hey, I changed all 'open' to 'show' in Bootstrap 4 beta 2 and didn't get the hover animation.
I changed bootstrap-dropdownhover.min.js and bootstrap-dropdownhover.min.css

Hello i changed it and make a repository use this but do not use .min.js just use js file with css and it will work in bootstrap 4 .
https://github.com/hassanbinsafdar/bootstrap-4.0-multilevel-dropdown-hover


Network Plugins: Earlier all the Network Plugins for Pod Network was part of Kubernetes.io sub-pages. Now the installation is in a different page - major issue for CKA Aspirants

Network Plugins: Earlier all the Network Plugins for Pod Network was part of Kubernetes.io sub-pages. Now the installation is in a different page - major issue for CKA Aspirants

It will be great the bring back the same structure back to kubernetes.io, so CKA aspirants can get get the right command and install instructions under kubernetes.io.

This will be very helpful and needed immediately - I am appearing for CKA on 6th Feb 2022, and I have no clue how to deal with Pod Network plugin installation command.

Kindly review the page:
https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/network-plugins/
`Network Plugins
Network plugins in Kubernetes come in a few flavors:
CNI plugins: adhere to the Container Network Interface (CNI) specification, designed for interoperability.
Kubernetes follows the v0.4.0 release of the CNI specification.
Kubenet plugin: implements basic cbr0 using the bridge and host-local CNI plugins`
Kindly note the pages are linked to GitHub - which are not allowed to be accessed during CKA Exam.
Trust this will be helpful for triaging.


Actions: create a Lint action that catches common silent build mistakes

This would be a nice feature advertisement for blight's functionality.
Some things we could lint for:

-DFORTIFY_SOURCE instead of -D_FORTIFY_SOURCE: the former is a no-op, meaning that the user's expected fortifications are not actually compiled in.
-D_FORTIFY_SOURCE without -O or higher: fortification doesn't work with -O0.
Flags that disable baseline mitigations (-fno-stack-protector, -z execstack, others?)

H/T for @disconnect3d for the idea + initial lint ideas.

Fwiw:

-z noexecstack works while -z noexecstackLOL will only say: /usr/bin/ld: warning: -z noexecstackLOL ignored. but will pass; same for -Wl,-z,noexecstackLOL


Need to incorporate folding/regex feature into individual test results

Currentlly only showing folded sections in Full Output. This is because the Folded Output section is the only section to which the folding algorithm applies. Need to also apply to each individual test cases result.

Implemented; should be avail next release (2.1.0)


[Proposal] Support for 120Hz+ monitors

I've recently started using FramerJS to prototype gestural interaction applications with large screens (I promise to post a video on FB once there is something ready to show). So far I'm loving it, but I found it weird that in TVs or monitors with a high refresh rate (120Hz or more) all the animations get way slower. Around 2x slower.
It could be related to this: http://lists.w3.org/Archives/Public/public-web-perf/2013Aug/0057.html
My question: would it be possible to add a global property to a Framer object (maybe a speed property to the Animation class?) so that the developer can instantly adapt animations to these type of devices?

Hey, interesting. You can try and set the Framer loop to half or twice the speed:
https://github.com/koenbok/Framer/blob/master/framer/AnimationLoop.coffee#L26
Framer.Loop.delta = 1/60/2
Framer.Loop.delta = 1/60*2

Let me know how that goes! I'm curious :-)

That's exactly what I was looking for, thanks for the quick answer!
Today I was making more tests with different resolutions (4K, FullHD), and cables (HDMI and DVI) but it seemed to work just fine -i.e. without any delta modifications-. It's very strange, as I'm 100% sure that all the animations were delayed the day before, even when loading regular sites that use requestAnimationFrame().
In any case, the graphic performance (even on 4K) is outstanding, Framer just keeps the pace at 60fps at very high screen resolutions! My next goal is to check how it behaves with multiple image loading + HD videos.
I'll make sure to send you a proper update with the results :)


Support ScanCode input bindings

What problem does this solve?
Bind to actions based on the scancode sent. These correspond to the semantic values of keys, not just their position on the keyboard.
As a result, these are essential to supporting non-US keyboards (and for text entry).
What solution would you like?
Add an InputKind variant that supports this.
[Optional] How could this be implemented?
In order to handle this correctly, we'll need to parse the keyboard event streams themselves.
[Optional] What alternatives have you considered?
Assigning semantic value is out of scope for a first attempt; ideally this is handled at the winit level: https://github.com/rust-windowing/winit/issues/1904
Related work
See https://github.com/bevyengine/bevy/issues/2052 and https://github.com/bevyengine/bevy/issues/862 for upstream work in Bevy.
More discussion in winit: https://github.com/rust-windowing/winit/issues/753

See also: https://github.com/bevyengine/bevy/pull/5495


Use Jenkins 2.361 LTS

Update to 2.361 LTS (context).
:warning: The Jenkinsfile needs an update by someone with write permission, eg. using the example.

[x] Make sure you are opening from a topic/feature/bugfix branch (right side) and not your main branch!
[x] Ensure that the pull request title represents the desired changelog entry
[x] Please describe what you did
[x] Link to relevant issues in GitHub or Jira
[x] Link to relevant pull requests, esp. upstream and downstream changes
[x] Ensure you have provided tests - that demonstrates feature works or fixes the issue

ping @timbrown5
I have updated the PR to the current recommended version (2.361.4).

Superseded by #83.


golang code

Thanks, go added.

It will be good to add main function to show case it, and an example to loop the colors (name and its hex colors)

merged, you can update, add the main function by another PR.
On Sat, Sep 3, 2022 at 8:42 PM Max Base @.***> wrote:

Merged #11 https://github.com/BaseMax/PersianColors/pull/11 into main.
—
Reply to this email directly, view it on GitHub
https://github.com/BaseMax/PersianColors/pull/11#event-7317451663, or
unsubscribe
https://github.com/notifications/unsubscribe-auth/AAUI56G473OQKST7ODXB2E3V4OLZFANCNFSM6AAAAAAQD73SMI
.
You are receiving this because you are subscribed to this thread.Message
ID: @.***>

Well, I'll add it


feat: add handling functions for nonce mismatches

Overview
This will be required for automagically handling nonce mismatches by the signer
Checklist

[ ] New and updated code has appropriate documentation
[ ] New and updated code has new and/or updated testing
[ ] Required CI checks are passing
[ ] Visual proof for any user facing features like CLI or documentation updates
[ ] Linked issues closed with keywords

Codecov Report

Merging #2235 (ea8d65f) into main (29f7f50) will decrease coverage by 0.01%.
The diff coverage is 45.45%.

@@            Coverage Diff             @@
##             main    #2235      +/-   ##
==========================================
- Coverage   21.53%   21.53%   -0.01%
==========================================
  Files         127      128       +1
  Lines       14335    14346      +11
==========================================
+ Hits         3087     3089       +2
- Misses      10945    10951       +6
- Partials      303      306       +3

Files Changed
Coverage Δ

app/errors/insufficient_gas_price.go
75.00% <ø> (ø)

app/errors/nonce_mismatch.go
45.45% <45.45%> (ø)

... and 1 file with indirect coverage changes


Update ScrollView after notification closes in Editor Layers View

The code in this PR fixes ScrollView not being updated in Editor's Layers View when notification closes after adding a layer, leaving an unexpected scrollbar where scrolling is not possible because of a wrong height.
We need to trigger a ScrollView update because the notification DOM node is not a child of the mentioned ScrollView, so perfect-scrollbar does not know that a layout change has been made.
Changes proposed in this pull request:

editor/editor-pane.js

Inject updateScrollWhenExist function from ScrollView to be available in LayersView instance

editor/layers/layers-view.js

Add updateScrollWhenExist to LayerView required options
Listen to notification:close event
Trigger ScrollView update on notification:close event handler

Current behaviour after fix

Questions

Do you want me to add the test to a new describe section in order not to do Notifier setup and teardown before and after each test and do it only in the notification test?

Fixes: #12953

Can one of the admins verify this patch?

Can one of the admins verify this patch?

I have just updated the PR with the changes.
@matallo You are right. I didn't notice that variable below. Consistency is king.  :)
@ivanmalagon I'm afraid it wouldn't work using self (I tested it) because this in that scope refers to the ScrollView instance. The variable assigned to self is the EditorPane instance in line 62, and the EditorPane doesn't have any method named updateScrollWhenExist.
But I agree with you that it is tricky for anyone to read, so I think that it would be better to assign this to a variable named scrollViewInstance, so anyone that reads it know what it is happening there. I have updated the PR with that change; please let me know what you think.
I tried to make it easier when I thought the best way to achieve the ScrollView update, but I didn't find any more readable and straightforward way to inject the function from the ScrollView to the LayersView :(

Why this was not applied?

It was the code test that I did prior joining CARTO 😂

Can we merge it?

I'm taking this one

It is working OK (broken CI because of the external fork)


read allocates memory for Rasters that have already been read

The documentation for read states read will move a Rasters.jl object completely to memory, which seems to imply that its intention is to read Rasters from disk into memory. However, I've noticed that calling read on a Raster that has already been fully loaded into memory will create a duplicate array, essentially acting like copy.
This behaviour can be quite annoying when I simply want to ensure that a Raster has been loaded before applying some sort of operation, since it results in unnecessary allocations. I'd like to propose that read should simply return the given Raster as-is when its contents have already been loaded. Alternatively, it would be good to have a method that implements this behavior.

Hmm yes probably doing nothing is better if the object is already in memory.
I guess the rule should be if its a disk array read it, otherwise keep whatever it is?


support for Atom (or any other text editor)

Currently, the Journal program opens the file in Sublime Text only (program calls are here and here).
This should be changed to enable support for configuring the journal to open in Atom, Vim, or any other text editor (except Emacs, of course).

Good idea! I have a branch to implement support for atom. The only issue may be that different editors will open to specific line numbers differently/not at all. I haven't looked into editors aside from atom and sublime.


How do we keep track of spent silentpayments?

Right now we can track received silent payments, but we can't tell when they have been spent.

Moved to https://github.com/cygnet3/sp-backend/issues/1


Full-app re-render

Version

"react-router": "^6.0.0-alpha.3"
"react-router-dom": "^6.0.0-alpha.3",

Please replace this sentence with the React Router version that you are using.
Steps to reproduce
export default () => {
  return (
    <Root scheme={scheme} theme={muiTheme}>
      <Header />
      <Sidebar />
      <Content>
        <React.Suspense fallback={<Loader />}>
          <BrowserRouter>
            <Routes>
              {routes.map((route) => (
                <Route
                  key={uid(route)}
                  element={<route.component />}
                  path={route.path}
                />
              ))}
            </Routes>
          </BrowserRouter>
        </React.Suspense>
      </Content>
      <Footer />
    </Root>
  );
};

Expected Behavior
On my application, I have a set layout, Header, Sidebar, Footer and Content . Obviously most of the UI does not change, namely, the Header, Sidebar, Footer, the only layout part that changes according to the routes is my Content. However, when I go from route A to route B, the entire application gets re-render, this is absolutely not the expected behaviour.
Actual Behavior
The question (bug?) here is how to not cause the entire application to re-render but only the section within the Content? I am not sure if it is possible on a location change to not trigger an application re-render, maybe with some client-side routing?

When it is a bug, then it is not a bug of RR. RR only takes control of everything below the <BrowserRouter /> component.

@MeiKatz the issue happen in the root application component below the <BrowserRouter> component, changing the route should not trigger the entire application to re-render, am I wrong?

@aress31 You're right, it shouldn't, but the way React works there is nothing RR could do to cause such behaviour.

So this is the expected behaviour and can't be fixed, am I getting this right?

@aress31 There might be a solution but I think RR is the wrong location for it.

@MeiKatz is correct. React Router isn't causing this.


Gray lines will be shown for each page

http://gyazo.com/5ab6e31688e83ce388ce00dec245c6b8
win7 chrome

I found reason: It calculates window.outerWidth / window.innerWidth  to get browser zoom setting. chrome is not maxium, window.outerWidth contains window's border. So, calculated value is bigger than actual zoom setting.


Events

Continuation of https://github.com/zendesk/cross-storage/pull/32

CI wasn't running with the previous PR, so let's see if this works in all browsers.

@alvinlindstam Looks like the storage events don't quite behave as expected in IE

@danielstjules Ah, in what way? I'd be happy to assist in fixing it if possible.

@alvinlindstam Here's a link to the job associated with your original PR: https://travis-ci.org/zendesk/cross-storage/builds/242063000 The squashed commit is here: https://github.com/zendesk/cross-storage/pull/41/commits/360b8962858b33b9eb0341ef39f1c8d73576d821
IE in particular is firing more events than intended

Ah, I guess it's related to the known issues mentioned on https://caniuse.com/#feat=namevalue-storage.

The "storage" event is completely wrong in IE:
IE10 : The storage event is fired even on the originating document where it occurred. Causes problems with multiple windows websites, and huge problems with iframes.
IE11 : The storage event's oldValue and newValue are identical (newValue is supposed to contain the storage's updated value).
Partial workaround: regularly probe the storage's value and compare with the last known value on all pages where you want to listen to this event, and track the last submitted value to determine if the modification was triggered locally.

Should we try to use their suggested workaround or document the bugs? The same issues would occur if using vanilla localStorage on the same host with IE.
The workaround would be more difficult to do on the client side however, so if it should be worked around it should probably be on the hub side. But continuously polling and storing all key-value-pairs in localstorage could be quite a performance issue (mainly memory) depending on the stored data.

Is this feature still planned?
Regarding this:

continuously polling and storing all key-value-pairs in localstorage could be quite a performance issue (mainly memory) depending on the stored data.

Maybe this could be an opt-in feature so that this cost is not incurred unless you intend to use events?


Admin Interface: Todo

On project and user profile page:

[ ] update session information in local storage: add project's shortcode, where the user is admin
[ ] is logged-in user sysAdmin or projectAdmin? if false: don't load project members and groups
[ ] in case of project admin: he's not allowed to see all users and to add one of them to the project --> talk with Ivan
[ ] disable collaboration, groups and data model tab for "normal" project members
[ ] fix gravatar image size by adding s=256 to the url; this should be fixed in @knora/action module
[ ] add gravatar to the user menu
[ ] edit user profile in the sidebar and remove edit user form from account
[ ] edit user profile (including update password) in dialog box; will be used in collaboration
[ ] fix password form
[ ] project search field in list of projects

 Kuirl: admin interface

Password form: Fix warning
[DOM] Password forms should have (optionally hidden) username fields for accessibility: (More info: https://goo.gl/9p2vKq)


Unable to establish connection between micro structure and sugar or phopshate (and partially - unresolved monomer)

Steps to Reproduce

Go to Molecules mode
Put Benzene ring on the canvas
Switch to Macro mode - Flex
Put sugar, phosphate and unknown monomer
Or load from file: Unable to establish connection between micro structure and sugar or phopshate (and partially - unresolved monomer).zip
Try establish connection between them and any atom of ring

Actual behavior
It doesn't work
https://github.com/user-attachments/assets/595428f9-ef06-42cd-b9af-6a45d1098f09
Expected behavior
It works
Versions

Ketcher Version 2.26.0-rc.1 Build at 2024-10-04; 18:45:04
Indigo Toolkit Version 1.25.0-rc.1.0-g704673fcb-wasm32-wasm-clang-19.0.0
Chrome Version 129.0.6668.90 (Official Build) (64-bit)
Win10

Issue found while testing: https://github.com/epam/ketcher/issues/5359

Verified.

Ketcher Version 2.26.0-rc.6 Build at 2024-10-25; 09:57:12
Indigo Toolkit Version 1.25.0-rc.4.0-g8862ccb7e-wasm32-wasm-clang-19.0.0
Chrome Version 130.0.6723.59 (Official Build) (64-bit)
Win10


Initial files for move to cloudfoundry

What? and Why?

Initial files for move to cloudfoundry - note port 5000 in Procfile

Checklist

CHANGELOG.md updated?

 2nd 1/2 SDX into Cloudfoundry (5)

Codecov Report

Merging #132 into master will increase coverage by 0.02%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master     #132      +/-   ##
==========================================
+ Coverage   79.11%   79.13%   +0.02%
==========================================
  Files          17       17
  Lines         972      973       +1
==========================================
+ Hits          769      770       +1
  Misses        203      203

Impacted Files
Coverage Δ

transform/settings.py
100% <100%> (ø)
:arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 733cd8e...240f2dd. Read the comment docs.


Add ignore rule for pycache directories in default python template

This is a very simple change, but basically the __pycache__ directory should be ignored by default for python projects. This is most noticeable looking at the filesystem graph for a python project, which will contain directories as leaf-nodes (since none of the files in that directory have a .py extension). Here's an example running emerge on the emerge repo without this ignore rule:

Note that all the blue leaf nodes in this image are from __pycache__ directories.

@tonysyu Again: thanks for another useful addition like this. That really motivates to keep working on this project 😉. Will create a new release including all your fixes and additions soon 🚀.


Check for ENV vars that interfere with local testing

Help us discover local state problems that interfere with releases
Checklist

[X] I've run bundle exec rspec from the root directory to see all new and existing tests pass
[X] I've followed the fastlane code style and run bundle exec rubocop -a to ensure the code style is valid
[X] I've read the Contribution Guidelines
[X] I've updated the documentation if necessary.

Could we unset them instead (and maybe warn the user?)

@lacostej Either is probably fine, but I think my preference is make our mistakes noisy, rather than ignorable.


refactor: split models into separate files

In PR37 noticed, the model.rs is growing rapidly, so its hard to deal with.
This commit refactors the model structure to use separate files per concern.
No logic changed.

@BerSomBen can you please resolve the conflicts that occur after merging #37


docs: update description of hook vars, note naming expectations of default policy

Copied from #84
Fixes #77

v1.33.0 has been just released.

v1.33.0 has been just released.


Remove wrapping transaction for single read api

IMPORTANT: All PRs must be linked to an issue (except for extremely trivial and straightforward changes).
Fixes #2552
Description
Removes wrapping of db transaction for single resources reads to prevent potential blocking
Alternative(s) considered
Have you considered any alternatives? And if so, why have you chosen the approach in this PR?
Type
Choose one: (Bug fix | Feature | Documentation | Testing | Code health | Builds | Releases | Other)
Screenshots (if applicable)
Checklist

[ ] I have read and acknowledged the Code of conduct.
[ ] I have read the Contributing page.
[ ] I have signed the Google Individual CLA, or I am covered by my company's Corporate CLA.
[ ] I have discussed my proposed solution with code owners in the linked issue(s) and we have agreed upon the general approach.
[ ] I have run ./gradlew spotlessApply and ./gradlew spotlessCheck to check my code follows the style guide of this project.
[ ] I have run ./gradlew check and ./gradlew connectedCheck to test my changes locally.
[ ] I have built and run the demo app(s) to verify my change fixes the issue and/or does not break the demo app(s).

thanks for this @LZRS!


chore(RHTAPWATCH-376): segment-bridge member-cluster component

Deploy a component for the User Journey effort as segment-bridge. This is needed in order to have a Service account with read permissions to the Namespaces resources from the member clusters. Those resources will be used in a script that'll map between events and cluster's owner

/retest-required

/retest-required

/retest-required

/retest-required

/retest-required

Since there's a chance the toolchain namespace won't be set in development (due to a flag configuration), leading to a sync issue, and as we don't need the development environment in the foreseeable future for this component - I've added the component to the delete overlay of development

/lgtm


Add handling of total flex consumption for grid operator

Description
Add handling of total flex consumption for grid operator

Codecov Report

Merging #78 (5e58205) into main (1ab3ad5) will decrease coverage by 0.71%.
The diff coverage is 77.45%.

@@            Coverage Diff             @@
##             main      #78      +/-   ##
==========================================
- Coverage   81.48%   80.78%   -0.71%
==========================================
  Files         332      437     +105
  Lines        6175     9929    +3754
  Branches      579        0     -579
==========================================
+ Hits         5032     8021    +2989
- Misses        951     1570     +619
- Partials      192      338     +146

Impacted Files
Coverage Δ

...s/Aggregations/AnAggregationProcessHasCompleted.cs
100.00% <ø> (ø)

...actions/Aggregations/RetrieveAggregationResults.cs
100.00% <ø> (ø)

...Transactions/Aggregations/SendAggregationResult.cs
57.62% <0.00%> (-28.58%)
:arrow_down:

...ce/Domain/Transactions/Aggregations/Aggregation.cs
8.33% <0.00%> (ø)

...reData/NotifyAggregatedMeasureDataMessageWriter.cs
96.61% <ø> (+1.69%)
:arrow_up:

...ansactions/Aggregations/AggregationResultMapper.cs
0.00% <ø> (-7.50%)
:arrow_down:

...grationTests/Factories/AggregationResultBuilder.cs
100.00% <ø> (ø)

.../OutgoingMessages/AggregationResultMessageTests.cs
95.65% <ø> (ø)

...ransactions/Aggregations/AggregationResultTests.cs
100.00% <ø> (ø)

.../NotifyAggregatedMeasureDataDocumentWriterTests.cs
78.12% <ø> (ø)

... and 232 more

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


[MER-617] Fix format date times

https://eliterate.atlassian.net/browse/MER-617

Closing in favor of https://github.com/Simon-Initiative/oli-torus/pull/2156


Added minimum players option to $splitlobby

$splitlobby command can now be used with a minimum number of players.
Examples:

$splitlobby 4
Will pass only if at least 4 people agreed to split to the new lobby.
$splitlobby
Works as before (sets minimum number to 1).

In addition to this, fixed a bug where the person who calls  $splitlobby would always be moved to a new lobby even if no one else voted to move.

Minus the relatively small issues I've flagged up this is a really nice fix. I look forwards to merging it :)


[stable/dashboard] sidecar containers for dashboard

Is your feature request related to a problem? Please describe.
Many time I need to deploy the dashboard with a sidecar container (ex: an auth0 proxy like this https://github.com/trondhindenes/Kubernetes-Auth0 )
Describe the solution you'd like
I want to have the option to specify a sidecar container as a configurable parameter.
Describe alternatives you've considered
No alternative
Additional context
Please take a look to Prometheus Helm chart:
https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml#L508
https://github.com/helm/charts/blob/master/stable/prometheus/templates/server-deployment.yaml#L137-L139
Ref: https://github.com/kubernetes/dashboard/issues/4355

That would be very useful for me as well!

I'd love to have this feature too!

@ChiefAlexander or @taharah do you think that is it doable? Make sense for you?

@kfox1111 or @desaintmartin do you think that is it doable? Make sense for you?

Yes, it would! I think the implementation should be as generic as possible to support other sidecar use cases.

I understand that if I want this thing I'll have to do it myself 😞

Basically, yes, unless you want to pay someone to do it.

I will do it in the next future...

Not now... in the future :P

I'll do, I promise! 😝

But not now :P

I don't have time now 😛

Moved the feature request here https://github.com/kubernetes/dashboard/issues/5125


Initializing runtime without entry point function and command line arguments

While trying to integrate HPX into an existing simulation framework, I ran into the issue of initializing the runtime while providing neither an entry point function nor command line arguments. Due to the specifics of our framework at the point where HPX should be initialized I cannot access argc/argv, as the corresponding code is located in a module loaded dynamically at runtime. Additionally, it is necessary to suspend/resume the runtime in our application.
Following the example in the corresponding section of the manual, I could not find an overload for hpx::start() or hpx::init() which allows me to initialize the runtime as described above while ideally passing the configuration to HPX via a vector of strings.

Thanks @severinstrobl, this may just be an oversight on our part, but should be easy enough to add. I'll try to get to this next week. If you'd like to have a go at it the overloads would be missing from hpx/hpx_init.hpp and hpx/hpx_start.hpp.

We do have such an overload for hpx::init and hpx::start:

https://github.com/STEllAR-GROUP/hpx/blob/master/hpx/hpx_init.hpp#L539-L540
https://github.com/STEllAR-GROUP/hpx/blob/master/hpx/hpx_start.hpp#L550-L551

The vector can contain either configuration settings or verbatim command line options. For instance:
{
    std::vector<std::string> cfg = {
        "--hpx:threads=1",
        "--hpx:print-bind"
    };
    hpx::init(cfg);
}

If I understood @severinstrobl correctly he's looking for an overload for nullptr_t  (no entry point) and a configuration vector, which is actually missing. I don't remember now if we support passing in 0 and nullptr, for argc and argv, respectively, but that might either already work or be something we could add as well.

Exactly, I did not want to pass an entry point. Passing 0 / nullptr for argc / argv to start is prevented by an assertion: https://github.com/STEllAR-GROUP/hpx/blob/8f9678cae0b4e5902582e4f88855d69634beeb57/hpx/hpx_start_impl.hpp#L361

@severinstrobl So IIUC, you are asking for an initialization function that explicitly inhibits running a startup-function but still allows to pass in configuration strings and command line options through a vector<string>:
{
    std::vector<std::string> cfg = {
        "--hpx:threads=1",
        "--hpx:print-bind"
    };
    hpx::init(nullptr, cfg);
}

Is that correct?

@hkaiser If I'm not misinterpreting the docs I think for what I'm trying to do I need an overload of hpx::start, rather than hpx::init.
While this works for my test case, there seems to be an (unrelated?) issue with --hpx:print-bind: When I provide this option via the configuration vector, it seems to be ignored.

If I'm not misinterpreting the docs I think for what I'm trying to do I need an overload of hpx::start, rather than hpx::init.

Sure. I'd add overloads for both functions anyways.

there seems to be an (unrelated?) issue with --hpx:print-bind: When I provide this option via the configuration vector, it seems to be ignored.

This is definitely unrelated, I'll look into this.


[kubernetes] Update 1.21.13

https://github.com/kubernetes/kubernetes/releases/tag/v1.21.13
Draft because auto workflow should handle this...

Reached release-data, so should be reflected today (🤞🏽) https://github.com/endoflife-date/release-data/commit/d20c8c3d283f8fb4c1ffa255a4dbaa7730b1a056

@BiNZGi this PR still looks like Draft , will you have anything to add or will we merge this with current shape ?

Updated already: https://endoflife.date/kubernetes with #1253 so closing this.


Support for single reinstall in Jest multi worker mode

Is your feature request related to a problem? Please describe.
Carrying on from the discussion here: https://github.com/wix/Detox/issues/1279
Describe the solution you'd like
Detox should only install the app once per worker instead of reinstalling for every test file.
Describe alternatives you've considered
There's some workarounds using --reuse, but they only work with --runInBand or --workers 1.

@rotemmiz The workaround with --reuse could work, but there's some caveats. With Jest, the global beforeAll and afterAll hooks will also run for each worker, so that needs to be taken into account since users will have detox.init() in beforeAll as well as adapter.afterAll() and detox.cleanup() in afterAll.

I have some ideas on this but I need to conduct my own investigation.

detox test --reuse worked for me

Hey, we managed to solve this based on jest-environment-puppeteer implementation
What we did:

Spawn n amount of detox web-socket servers, where n is the amount is the number of jest workers specified
Pass generated session (web-socket-server-url and uuid) to each test suite with the process.env hack (which should be easy to change when jest add a new API to pass primitives https://github.com/facebook/jest/issues/7184#issuecomment-492122367)
On each test suite, initialize detox with session generated from step 1

Proof of concept: jest-environment-detox
What do you think? @rotemmiz

@noomorph The solution by @sbycrosz works really well! Any way we can integrate it into detox itself?

@hammadj, must have been fallen between the cracks, sorry. Looks interesting, I'll look closer.

Can't look in the nearest future, sorry, maybe on Friday but not likely. Got many other tasks at the moment.

@noomorph No worries, we are using it in our project successfully, no rush to integrate right away. Thanks for your attention and support on this!

Hey, can we integrate this into detox? We found there are some changes in the internal APIs that prevents the workaround from working 😢

@sbycrosz could you elaborate on these changes? @noomorph what's your opinion on this?

Hello!
We found that on the version 15.1.4, you also need to pass in both artifactConfig along with deviceConfig object to spawn a Detox server.
So we'll need to change https://github.com/stashaway-engineering/jest-environment-detox/blob/master/lib/global.js#L25
const detox = new Detox({ deviceConfig });

to
const detox = new Detox({ deviceConfig, artifactConfig });

Cheers!

Seems like still an issue! has there been any updates on this ?
I was trying mocha but it is relaunching app in every test file.
It would be helpful if someone gave minimal example

We plan to close this issue in Detox 20 after one more heavy refactor.

@noomorph but is there any workarounds to get it to work right now ?

@ShivamJoker I don't think there are easy ones.
@d4vidi the question made me wonder if we can adjust --reuse behavior so that it installs the app if it is missing on the device. This way, users can (at their own risk) have a scripting workaround for this issue:

first test does not find the app on the device, and installs it
the consequent ones reuse the app
in globalTeardown, there can be a custom teardown script that goes over all the booted devices and runs a hard-coded uninstall app.bundle.identifier...

WDYT?

We'll try to take it in Detox 20, but not promising...

Well 1st thing's 1st, I don't understand the use case for --reuse other than for local dev time, as long as reinstalling is the only means for cleaning things up between tests/suites/files. So with that, I'm inclined to say that #2534 is the only optimization we need at the moment. Maybe we ought to put our efforts into doing something similar for iOS, rather than in more easily enabling discouraged usage patterns...

This will be addressed under the DAS project (device allocation will include app install - at least by default), and no sooner.

This aligns with our future vision for the DAS project. I see no point in keeping it here as well.


Third-Party GPS Deprecated

The "GPS data settings" in this article refers to a number of fields that are labeled as "Deprecated" within the actual application. This documentation should be updated to either remove this section if this capability is no longer supported or with instructions on how to use the replacement functionality.
Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: f6c4cc24-2dd8-908a-aee2-3b52360eeaa7
Version Independent ID: 0dfc2d9a-ad3d-e7dd-3a7a-38da87565272
Content: Configure default settings (Dynamics 365 Field Service)
Content Source: ce/field-service/configure-default-settings.md
Service: dynamics-365-field-service
Product: dynamics-365
GitHub Login: @FieldServiceDave
Microsoft Alias: daclar

Thank you for sharing this feedback. I started working on Field Service docs a few days ago and do some housekeeping on open issues.
I looked through the commit history of the article and the GPS Data section was removed in September 2021.
Closing this issue as solved. Please keep your feedback coming!
Thanks,
Michael


CLEAN:  remove pandas.compat

pandas.compat is not public, so removed it (and various references to it)

Thanks, @Dr-Irv !


running uchiwa as root in order to bind to ports < 1024

if we don't run as root we have to go 'the extra mile' in order to bind to port 443 or 80 which are standardised to be used to e.g. ssl connection
if you want to make your dashboard accessible to a wider audience it is clearly useful to use standard ports for this in order for people to be able to use it.
since uchiwa is capable of SSL on it's own, this has gotten way more important than before

Thank you for your pull request but I firmly believe you should run Uchiwa using a non-root user account for obvious security reasons and this small loss of usability is largely compensated by the gain of security. For this reason, I'll have to refuse it. Thanks


fixes  issue #2 Race condition by adding an optional createCollectionCallback

I fixed  issue #2 Race condition by adding an optional createCollectionCallback that is called after the MongoDb Client has connected, the collection created, and the index created.
This is similar to https://github.com/hotelde/node-cache-manager-fs fillcallback option.
Sample usage added to README.md:
const mongoCachePromise = () => {
return new Promise((resolve) => {
const mongoCache = cacheManager.caching({
store: mongoStore,
uri: 'mongodb://localhost:27017/nodeCacheDb',
options: {
host: '<IP_ADDRESS>',
port: '27017',
    // username: 'username',
    // password: 'pass',
    database: 'nodeCacheDb',
    collection: 'cacheManager',
    compression: false,
    server: {
      poolSize: 5,
      auto_reconnect: true,
    },
  },
  createCollectionCallback: () => {
    console.log('done creating collection');
    return resolve(mongoCache);
  },
});

});
};
mongoCachePromise.then((mongoCache) => {
return mongoCache.wrap(key, () => {
return getUser(userId);
});
});

@gofreddo  thx for pull request.  This weekend i check code and if all works fine me


Resolve failing tests

The failures in the build of current master stem from lack of warning message, generation of which is recovered here.

Ok let's just sneak this into master to fix the failing build...


Laravel 4.2 and composer = incompatibility

Installing via composer causes the following error in Laravel 4.2
Problem 1
- Installation request for edvinaskrucas/notification 4.* -> satisfiable by edvinaskrucas/notification[4.0.0].
- edvinaskrucas/notification 4.0.0 requires illuminate/support 5.* -> no matching package found.

Version 4 and up uses Laravel 5, for laravel 4.2 use 3.*

well than this should be documented in the Readme.md since it still says:
Notification package for Laravel4


Add header photo to top of webpage

having just previewed this, it may be better for the photo to be the same width as the text box below it (A communal workshop and collective of... etc) but i'm not sure how to code that change in. A line break with the blue header above it may also look better

I've made some changes to this pull request.

I've made the hero image you provided be in the background of the text. This is what the website was originally designed for, but no one came forward with a suitable image at the time so it was left as a grey box.
I have moved all of the images into this repository so they are packaged with the website code and will also be hosted in the same place. This will greatly improve performance and make it easier to maintain the website in the future.

@James-BuildBrighton I forgot to say, let me know if you're happy with these changes and I'll make them public. The preview link above will have updated to reflect the changes.

@matt-oakes  thanks looks good! only minor things i'd mention would be if possible, to have a line break above the image to separate it from the blue navigation bar at the top (like that box was before), and if there's any way to avoid cutting out the bottom of the image/the bottom of the G in Brighton? I'm aware that may be due to the re-scaling to fit in the text box, so if not, just the line break will be fine and it's good to go


mavlink: fix use of -x flag

The -x flag did not do what it was supposed to do, namely enabling FTP.

@dagar CI failure looks like a false positive from clang-analyzer:
/home/travis/build/PX4/Firmware/src/modules/mavlink/mavlink_main.cpp:2301:4: error:
Use of memory after it is freed [clang-analyzer-cplusplus.NewDelete,-warnings-as-errors]

Thanks @dagar