Android tests are permafailing CI with diskspace issues

Probably we can add a hack to resolve this - but not sure what we want to remove to free diskspace

hey @phlax , is there a link to the failing jobs?  Is it the kotlin_helloworld one?  I see that one failing a lot (e.g. https://github.com/envoyproxy/envoy/actions/runs/6482932266/job/17603711441#step:5:6859), though the error message seems to be a download timeout (maybe it's timing out b/c it can't download due to no disk space?)
is the diskspace issue on the worker agents?  can we run du or something similar to see where the majority of disk space is taken up?  if it's old archives or test logs, i think we can just delete to free up the space

oh nevermind, i see it here: https://github.com/envoyproxy/envoy/actions/runs/6473012784/job/17574833005

can we just rm the /home/runner/runners/2.309.0/_diag/Worker_20231010-175424-utc.log file?  i'm not sure if i have login credentials to do so myself..

sorry, i meant rm /home/runner/runners/2.309.0/_diag/

yeah here is the ci log of fail https://github.com/envoyproxy/envoy/actions/workflows/mobile-android_tests.yml?query=branch%3Amain
re resolution - we have a toolshed action (git grep actions/diskspace) it has a parameter to tell it what to remove - it maybe the defaults remove stuff that is wanted - not sure
re _diag - not sure about that - re login it needs to be set in ci (not sure iiuc)

like in existing code
https://github.com/envoyproxy/envoy/blob/d5cf0c33430d858eba46eed2028e93ef5618b154/.github/workflows/codeql-push.yml#L28-L32
you may want to not remove what that is removing - the defaults are here https://github.com/envoyproxy/toolshed/blob/b974b20fbf9a2436c5c8643d5dfc6fdfcbbfe024/gh-actions/diskspace/action.yml#L2-L7
im not sure what is actually used by this ci - if it uses the build container then just remove the lot (ie use default)


Open a PR to live in public repositories

Your PR targeted the live branch, which is not allowed. The PR has been redirected to the default branch (main).

Docs Build status updates of commit f5e439d:
:warning: Validation status: warnings

File
Status
Preview URL
Details

csidevdocs-public/test.md
:warning:Warning

Details

.openpublishing.publish.config.json
:white_check_mark:Succeeded

csidevdocs-public/docfx.json
:white_check_mark:Succeeded

csidevdocs-public/test.md

Line 1, Column 1: [Warning: author-missing - See documentation] Missing required attribute: 'author'. Add the current author's GitHub ID.
Line 1, Column 1: [Warning: title-missing - See documentation] Missing required attribute: 'title'. Add a title string to show in search engine results.
Line 1, Column 1: [Warning: ms-author-missing - See documentation] Missing required attribute: 'ms.author'. Add the current author's Microsoft alias.
Line 1, Column 1: [Warning: ms-prod-or-service-missing - See documentation] One of the following attributes is required: 'ms.prod', 'ms.service'. Use ms.prod for on-premise products, or ms.service for cloud services.
Line 1, Column 1: [Suggestion: description-missing - See documentation] Missing required attribute: 'description'.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel

Docs Build status updates of commit f5e439d:
:x: Validation status: errors
Please follow instructions here which may help to resolve issue.

File
Status
Preview URL
Details

:x:Error

Details

[Error: PullRequestMergeConflict] Cannot sync git repo to specified state. Please make sure the pull request is mergeable.

For more details, please refer to the build report.
If you see build warnings/errors with permission issues, it might be due to single sign-on (SSO) enabled on Microsoft's GitHub organizations. Please follow instructions here to re-authorize your GitHub account to Docs Build.
Note: Broken links written as relative paths are included in the above build report. For broken links written as absolute paths or external URLs, see the broken link report.
Note: Your PR may contain errors or warnings unrelated to the files you changed. This happens when external dependencies like GitHub alias, Microsoft alias, cross repo links are updated. Please use these instructions to resolve them.
For any questions, please:Try searching the docs.microsoft.com contributor guidesPost your question in the Docs support channel


opt_clean reverses assignments

Here a minimal example:
module bsp_e(a, z);
	output z;
	input  a;
	wire s1;
assign
	s1 = ~a,
	z = s1;
endmodule

After executing the following skript:
read_verilog ../examples/bsp_e.v
proc; opt; fsm; opt; memory; opt
techmap; opt
dfflibmap -liberty ../libraries/yosys/mycells.lib
abc -liberty ../libraries/yosys/mycells.lib
clean
write_verilog -noattr ../tmp.v

I get the following output:
module bsp_e(a, z);
  input a;
  wire s1;
  output z;
  YNOT _0_ (
    .A(a),
    .Y(z)
  );
  assign s1 = z; /* this is reversed */
endmodule

I pinned it down to opt_clean reversing the assignments.
This happens with commit b226893461af46f2183be8ca9dfab62b49133c71

This is the intended behavior, and obviously a valid and useful transformation. Why do you think this is a problem?

Seems like the I was wrong...
anyway, when using the above skript with the LGSynth91 benchmark circuits, specifically s13201.1_synth.v, s35932_orig.v, s35932_sweep.v, s35932_synth.v, outputs without drivers are created
for s1320.1_synth.v it's signal g1017 and g6207 and for the s35932 circuits its signal g16496
I thought the example above was the reason for this...

I've downloaded LGSynth91.7z from http://ddd.fit.cvut.cz/prj/Benchmarks/, extracted
Verilog/smlexamples/s13207.1_synth.v and looked at g1017. Here is the entire portion of the file that drives this ouput:
module \s13207.1.bench_excerpt(g1017);
  output g1017;
  wire \[894] ;
  reg g1029;

  assign g1017 = g1029;
  assign \[894]  = g1029;

  always begin
    g1029 = \[894] ;
  end

  initial begin
    g1029 = 0;
  end
endmodule

The only thing that's ok about that code is the initial block, which can simply ignored because of the always assignment to g1029. (I'm not omitting a @* here. This is how it been in the original s13207.1_synth.v file! However, there was no Verilog standard in '91, so maybe it was usual practice back then to use something like that in synthesizable code.)
So g1017 is driven by g1029 which is driven by \[894], which is driven by g1029, ... This is just broken code. I see nothing wrong with the way Yosys handles that.
I'm now closing the issue. Feel free to post further comments if you find something that looks suspicious.

Even though I think that the behavior for this case is pretty much unspecified, I guess the expected behavior would be to not ignore the initial spec in favor of the "driver", because that driver is just a bunch of wires driving each other, and instead just set the whole thing to the init value. I've now implemented that in commit 4ea7054.


Create a Chinese (Simplified) translation of the 404 error page

Based on the English version of the 404 error page (https://github.com/devadvance/terminalcheatsheet/blob/staging/404.html), create a Chinese (Simplified) translation in https://github.com/devadvance/terminalcheatsheet/blob/staging/i18n/zh-CN/404.html.
You can copy this template to use a starting point: https://github.com/devadvance/terminalcheatsheet/blob/staging/i18n/new_language_templates/404.html
Look for TODO in the file for suggested places for translations. Please comment on this issue before starting work so people are informed that you are working on it.

I would like to work on this task.

@devadvance, changes done and PR submitted. Please check and let me know if anything I can help to close this issue.


Make past energy usage data optional

This solves #61 for now. If it turns out that the fields in question do not get readded we can remove them entirely.

Thank you for this contribution.
Tbh, I don't think this functionality is coming back to the get_energy_usage request.
I will follow up this PR with the deprecation of these four fields and the addition of the get_energy_data API, which should provide more or less the same functionality.

Thank you for this contribution.
Tbh, I don't think this functionality is coming back to the get_energy_usage request. I will follow up this PR with the deprecation of these four fields and the addition of the get_energy_data API, which should provide more or less the same functionality.

I just saw your latest commits. Looks good to me. I think deprecating the functionality doesnt hurt. (Btw. Thanks a lot for this awesome project <3)


DaddyLive only showing 184 channels

I just install cabernet and enabled the daddyLive plugin and the TV Guide plugin. I updated the channel list and EPG a few  times but DaddyLive only shows 184 channels. It just shows me channels from A-D. Is there anything I need to do to make it show the rest of the channels?

Your issue is and possibly caused by your system.  To determine the issue, you will need to turn on logging at the DEBUG level, then run the Channel update task for DL.  The log will show the list of channels it found and added. Provide the log portion that shows this.
Also, indicate if you turned on the EPG plugin in DL.

I have debug on but can't figure out how to see the log file. Can't find it I.the app config folder. Also I have all the pluins enabled except XUMO.

There are 2 log files.  The default is the STDOUT log file and is dependent on the system and install of the service.  As an example, on Linux, it would be in the syslog file.  The other log is in the data/log/ folder, but by default is disabled.  It sometimes does not log detailed exceptions, but is an easy way to access a log and not have other apps in it.  If you are having an issue with finding a log, recommend turning on the file log in data/log and use it.
Also, you did not answer the question. "indicate if you turned on the EPG plugin in DL"

Yes all the plugins are enabled including the EPG plugin. My data folder is empty even though I enabled debug on System Log Handler and File Log Handler.   To get an update, you need to "Reset Channel Data"  Go to Data Mgmt, Reset Edits: No, Plugin: DaddyLive. Do that so when you run the scheduled task, it will run.
Also, you did not answer the question. "indicate if you turned on the EPG plugin in DL"  I am NOT asking if you installed anything...

OK I reset the channel data for daddylive. Still getting 184 channels. Maybe I don't understand what you mean by "Turn on the EPG plugin".

To "turned on the EPG plugin in DL" go to settings > Stream Providers > DaddyLive and look at the "EPG Plugin to Use" setting.  If it is None, you are NOT using any EPGs.

On EPG Plugin to use I have it set to ALL

Docker logs are hard to read, but it says you already ran the updates, so it won't run again to keep people from spamming the task. To get an update, you need to "Reset Channel Data" Go to Data Mgmt, Reset Edits: No, Plugin: DaddyLive. Do that so when you run the scheduled task, it will run.

This fixed the issue. I just had to wait a long time.

First, install VLC on windows and load the channel.m3u file you get from cabernet into that.  Then try to play the channels there.  It will determine if its a plex issue.  If its a cabernet issue, you will need to provide a log of the stream.  Not sure how you did the docker container, but it needs port 5004 to stream.

I just looked and found that Deluge was using port 5004 for some reason, so I removed it from Deluge. Sadly I am still unable to play it on VLC and Plex. on VLC I get the error "Your input can't be opened:
VLC is unable to open the MRL 'http://<IP_ADDRESS>:5004/DaddyLive/watch/52'. Check the log for details." but for some reason I dont have anything on my data folder so can't see the log. And on Plex I get the error "Could not tune channel. Please check your tuner or antenna.". I updated the channels again and EPG but still can't stream anything.

Sounds like your docker container is blocking 5004.  Go to browser and type location http://<IP_ADDRESS>:5004/
If you get "501 - Not Implemented" then your port is open.  If not, then docker is your issue.

I am getting a "The connection has timed out" error. I'll spend some time today trying to figure out why docker would be doing that.

Where do I edit plex_docker_ip?

I just opened the channel.m3u with notepad and all the channels have the wrong ip. I changed one of them to my IP and its streaming. So gotta see how to change the IPs on all the channels on the m3u file.

I went ahead and did a fresh install and followed your directions and now I am able to play the channels on VLC. Even though I changed plex_docker_ip to my IP I am still unable to play anything on Plex but at least now I know its working, now I gotta play around with Plex and see whats the issue with it. Thanks for the help.


Support for Tiled Map

Adds the support to Tiled map format to the project and design the playground map for tests.
Tasks

[x] Add Tiled addon
[ ] Design the playgrond map


[Logging v2] Make strip underscores optional in journald logging config

Some logging endpoints (e.g. Elasticsearch) don't support fields with leading underscores.  The fluentbit config for journald logging should have the option to strip underscores.

We need to validate the config is properly set in the fluentbit config after chart install.
Edit YAML from Dashboard works, keeping the default false and also setting it as true
stripUnderscores: false
stripUnderscores: true
Validate this is true for the rke source. Validate the Elasticsearch keeps working with true and false.
Then sanity check this value is set with the rke2, k3s sources.

Rancher version: 2.6 master-head (07/14/2021) 58f6425
RKE2 v1.20.8+rke2r1 Imported using SELinux.
stripUnderscores: false did't add anything extra to the fluent-bit configuration and this default to Off according to the fluent-bit documentation. This is good.
stripUnderscores: true  adds the Strip_Underscores On line to the fluent-bit input configuration as expected. This is good too.
I checked the above on these scenarios:

Installing with stripUnderscores: false
Installing with stripUnderscores: true
Upgrading the installed chart with stripUnderscores: false
Upgrading the installed chart with stripUnderscores: true

I did a sanity check on an imported k3s v1.21.2+k3s1 cluster.
Configmap for k3s
rancher-logging-k3s
Configmap for rke2
rancher-logging-rke2


backport: Fail instead of truncate command name in CMessageHeader constructor

Cherry-picked from: b837b33
This removes a warning when compiling. The runtime assert can be safely used because:

This constructor is only used by CConman::PushMessage
All calls to CConman::PushMessage() are using NetMsgType constants.

forgot to mention the actual warning:
  CXX      script/libdogecoin_common_a-standard.o
In file included from /usr/include/string.h:519,
                 from serialize.h:20,
                 from netaddress.h:13,
                 from protocol.h:13,
                 from protocol.cpp:6:
In function ‘char* strncpy(char*, const char*, size_t)’,
    inlined from ‘CMessageHeader::CMessageHeader(const unsigned char (&)[4], const char*, unsigned int)’ at protocol.cpp:91:12:
/usr/include/x86_64-linux-gnu/bits/string_fortified.h:95:34: warning: ‘char* __builtin_strncpy(char*, const char*, long unsigned int)’ specified bound 12 equals destination size [-Wstringop-truncation]
   95 |   return __builtin___strncpy_chk (__dest, __src, __len,
      |          ~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
   96 |       __glibc_objsize (__dest));
      |       ~~~~~~~~~~~~~~~~~~~~~~~~~


How to change myself dataset？

How to change myself dataset？

Either prepare your own dataset in the same format as here https://github.com/lgalke/text_gcn/tree/master/data
or write your own data loading pipeline https://github.com/lgalke/text-clf-baselines/blob/a7363196a8b1ba730126e8b92dbbd1d45bfdd952/data.py#L50 (lines 50-97). The format is really just a list-of-documents where each document is a list of tokens and a list of class labels. In the example code above you can see how you can use huggingface tokenisers to prepare the list of tokens for each document.


Update Test Suite to match latest release features

We should start to rework the order of the steps to a release ...

https://github.com/oasp/ts-merger/wiki/releasing

can we access the public releases on npmjs by any download link as well such that we can download the packed js directly by maven?


Fix bug of disabling "Add" button used for adding subscribers on live update after subscribing.

We live update the "Add subscribers" UI on subscribing/unsubscribing
the stream and the field sub.can_add_subscribers is used to check
whether user is allowed to add subscribers or not.
The sub object used here is not the complete sub object and thus
sub.can_add_subscribers is undefined. We need to use the updated
sub obejcts with all the setting fields that we can get from
'stream_settings_data.get_sub_for_settings' before rendering UI.
This calculation of additional fields was done previously by
'stream_data.update_calculated_fields', but in d50462568ba9aa
call to 'updated_calculated_fields' was removed with the aim to
call 'get_sub_for_settings' before updating UI which was missed
in this part of UI update.
Fixes #18616.

Merged, thanks @sahil839!  @showell FYI.
I guess this is something where it'd have been really nice to have TypeScript, as then we could have the slim and full subscription objects have different types.


fix: can't load full classname Config file

Description
Fixes #5356
Checklist:

[x] Securely signed commits
[] Component(s) with PHPDoc blocks, only if necessary or adds value
[x] Unit testing, with >80% coverage
[] User guide updated
[x] Conforms to style guide

Sorry, I fixed the Rector errors.


Use more secure permissions.

Breaking change, isn't it? What's the motivation for this concrete change?

This was originally intended to match windows default permission. After using several other modules that depends this, I'm going to close this.


amazon ads SB campaign report

I get more data from the Advertising Console than from the Campaigns report
Can you tell me which api I am getting from which campaign report I am seeing on Amazon console, I am getting wrong campaign from version 2 of api and it has migrated to version 3 because I am getting the campaign from that api.

Hi @ADA-SaifShaikh, if you are seeing discrepancies between reports in the advertising console and reports in the API, I'd suggest opening a support ticket in Jira.


DDI 424 [Contextual Doc] Use "maintenance" over "legacy"

Amplitude Developer Docs PR
Description
Currently we use “legacy” in our dev doc for old version SDKs. It sounds too negative and customers usually don’t want to use it. But actually we still support bug fixes for these old SDKs. So after discussion, we agree to use “maintenance” over “legacy”.
Deadline
When do these changes need to be live on the site?
Change type

[ ] Doc bug fix. Fixes #[insert issue number]. Amplitude contributors include Jira issue number.
[ ] Doc update.
[ ] New documentation.
[ ] Non-documentation related fix or update.

PR checklist:

[ ] My documentation follows the style guidelines of this project.
[ ] I previewed my documentation on a local server using mkdocs serve.
[ ] Running mkdocs serve didn't generate any failures.
[ ] I have performed a self-review of my own documentation.

@amplitude-dev-docs

Since we are marking legacy SDK to maintenance.
Should we also update the GTM client-side legacy template as maintenance?

@yuhao900914 Marvin has the same question here. Love to hear what you think!


Home page Improvement review

#589

Looks good to me, thanks for the multiple commits, it is easier to review :+1:


060 releasenotes

[ ] Tag issue being addressed
[ ] Add tests
[ ] Update docstrings/documentation, if applicable
[ ] Update changelog

Codecov Report

Merging #454 into master will not change coverage.
The diff coverage is n/a.


chore: Address deprecated methods and types

Some deprecated types and methods have clear replacements but others are not documented at all.
Go through all of them and either finalize their removal or add a description of what steps need to be taken in order to finalize their removal.

@andrewvious


Installing the KubeArmorOperator throws an error when the "kubarmor" namespace is not created in the cluster

Bug Report
When I ran commands make, then make manifests, then make docker-build deploy from pkg/KubeArmorOperator the make docker-build deploy command throws an error,
GOBIN=/home/roopesh/go/bin go install<EMAIL_ADDRESS>/home/roopesh/go/bin/kustomize build config/default | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/kubearmorconfigs.operator.kubearmor.com unchanged
clusterrole.rbac.authorization.k8s.io/kubearmor-operator-clusterrole unchanged
clusterrole.rbac.authorization.k8s.io/kubearmor-operator-manage-controller-clusterrole unchanged
clusterrole.rbac.authorization.k8s.io/kubearmor-operator-manage-kubearmor-clusterrole unchanged
clusterrolebinding.rbac.authorization.k8s.io/kubearmor-operator-clusterrole-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/kubearmor-operator-manage-controller-clusterrole-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/kubearmor-operator-manage-kubearmor-clusterrole-binding unchanged
Error from server (NotFound): error when creating "STDIN": namespaces "kubearmor" not found
Error from server (NotFound): error when creating "STDIN": namespaces "kubearmor" not found
make: *** [Makefile:120: deploy] Error 1

When I ran the same command after creating the namespace "kubearmor" in the cluster, the command works fine by creating the resources in that namespace.
GOBIN=/home/roopesh/go/bin go install<EMAIL_ADDRESS>/home/roopesh/go/bin/kustomize build config/default | kubectl apply -f -
customresourcedefinition.apiextensions.k8s.io/kubearmorconfigs.operator.kubearmor.com unchanged
serviceaccount/kubearmor-operator created
clusterrole.rbac.authorization.k8s.io/kubearmor-operator-clusterrole unchanged
clusterrole.rbac.authorization.k8s.io/kubearmor-operator-manage-controller-clusterrole unchanged
clusterrole.rbac.authorization.k8s.io/kubearmor-operator-manage-kubearmor-clusterrole unchanged
clusterrolebinding.rbac.authorization.k8s.io/kubearmor-operator-clusterrole-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/kubearmor-operator-manage-controller-clusterrole-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/kubearmor-operator-manage-kubearmor-clusterrole-binding unchanged
deployment.apps/kubearmor-operator created

I would like to work on this issue if the maintainers want to proceed with it.

Hey @roopeshsn, thanks for opening this.
I think you can add a namespace.yaml in config/operator and include the same under resources block of kustomization.yaml
Please also proceed if you think there's a better way to tackle this.
Ping me if you've any questions.


Updates to the update-dask-sql.yml workflow

This PR updated the update-dask-sql.yml workflow. This should resolve the previous failures such as https://github.com/rapidsai/docker/actions/runs/5756838849/job/15606942103
Changes:

the-coding-turtle/ga-yaml-parser uses deprecated ::set-output, so switching to use yq & $GITHUB_OUTPUT instead. This requires using the rapidai/ci:latest container so yq is available.
Minor clean up for versions & PR text
Updates the checkout action version

/merge


fix backend dockerfile using venv

Description
This pull request addresses an issue with the Dockerfile for the backend of the Lipsync application. The issue was related to ensuring that the correct Python version (Python 3.7) is used for the wav2lip-hq component while keeping the Flask app isolated in its Poetry virtual environment.
Changes Made

Added Python 3.7 installation to the Dockerfile to create a Python 3.7 environment for wav2lip-hq.
Modified the installation of Poetry and backend dependencies to ensure they are isolated in the Poetry virtual environment for the Flask app.
Adjusted the entry point to run the Flask app with Poetry, ensuring that it uses the correct virtual environment.

Best fix is probably to downgrade flask to run on <python3.8. The specific python version used for wav2lip-hq unfortunately was not provided by the authors though, so its hard to determine which version it should be.


Add github action to build and optionally push on merge to master.

[x] Build on pull request to master
[ ] Build and push to vagrant cloud on merge to master

Merged in order to test merge to master action.

closes #11


Allow only collaborators to send commands

This has not been an issue for private repositories, where the access to
the PR comment thread is already limited, but for public repositories
this can be a real issue, if anybody can send the commands.
I suggest reviewing this diff with whitespace ignored. (with ?w=1 URL suffix on GitHub)

!merge

!check


Python3 pptk not found

Post installation, I ended up with  No module named 'pptk' error. I tried to install it with pip3 install pptk, however no matchin distribution found for pptk in pip3. I see in github page for pptk, there is a Python 2 version. There seems to be no Python 3 version:
https://github.com/heremaps/pptk/issues/39
Please advise.
In [2]: import kaolin as kal
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-2-679789fed074> in <module>
----> 1 import kaolin as kal

<frozen zipimport> in load_module(self, fullname)

/usr/local/lib/python3.8/dist-packages/kaolin-0.1.0-py3.8-linux-x86_64.egg/kaolin/__init__.py in <module>
     22 from kaolin import rep
     23 from kaolin import vision
---> 24 from kaolin import visualize
     25 from kaolin import metrics
     26 from kaolin import models

<frozen zipimport> in load_module(self, fullname)

/usr/local/lib/python3.8/dist-packages/kaolin-0.1.0-py3.8-linux-x86_64.egg/kaolin/visualize/__init__.py in <module>
----> 1 from .vis import *

<frozen zipimport> in load_module(self, fullname)

/usr/local/lib/python3.8/dist-packages/kaolin-0.1.0-py3.8-linux-x86_64.egg/kaolin/visualize/vis.py in <module>
     20 import kaolin
     21 import kaolin as kal
---> 22 import pptk
     23
     24

ModuleNotFoundError: No module named 'pptk'

what is your pip version ? Also Kaolin support 3.6.
I tried with pip==20.1.1 and python==3.6 and pip install pptk worked

I was using pip 20.0.2 with python 3.8.2. I have just upgraded to pip 20.1.1. Again no matching distribution found for pptk.

I was using pip 20.0.2 with python 3.8.2. I have just upgraded to pip 20.1.1. Again no matching distribution found for pptk.

Probably pptk did not release a package to Python 3.8 yet

@kunguz Try Python 3.7, it is best for all type of packages, it is very stable also.

Hi, thank you for using Kaolin! The library went through a major rewrite with the version 0.9.0, and only the core library components we can thoroughly test and maintain are now part of this repository. Please refer to our latest documentation:
https://kaolin.readthedocs.io/en/latest/
We are not relying on pptk anymore for visualization, please check our custom visualizer the Omniverse Kaolin App and Dash3D!
Closing the bug as it is no longer relevant to this repo.


Add an OTA download test to Darwin framework CI.

This tests that we can run all the way through an OTA update that takes multiple BDX blocks to transfer, and that the right data is received on the requestor side.

Fast-tracking platform-specific test change with platform owner review.


Connection refused

1:24:44 › gdown --id 1e3NtzpwBcUeakcfiYo_gB2IgxxFIGkVe
Traceback (most recent call last):
  File "/Users/zhaoxuyan/anaconda/bin/gdown", line 11, in <module>
    sys.exit(main())
  File "/Users/zhaoxuyan/anaconda/lib/python2.7/site-packages/gdown/cli.py", line 61, in main
    quiet=args.quiet,
  File "/Users/zhaoxuyan/anaconda/lib/python2.7/site-packages/gdown/download.py", line 52, in download
    res = sess.get(url, stream=True)
  File "/Users/zhaoxuyan/anaconda/lib/python2.7/site-packages/requests/sessions.py", line 521, in get
    return self.request('GET', url, **kwargs)
  File "/Users/zhaoxuyan/anaconda/lib/python2.7/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/zhaoxuyan/anaconda/lib/python2.7/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/Users/zhaoxuyan/anaconda/lib/python2.7/site-packages/requests/adapters.py", line 508, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='drive.google.com', port=443): Max retries exceeded with url: /uc?id=1e3NtzpwBcUeakcfiYo_gB2IgxxFIGkVe (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x106748f10>: Failed to establish a new connection: [Errno 61] Connection refused',))

Perhaps you need to change the permission to 'Anyone can access with link'.

I have this problem too. I changed the permission as you said,but it still can't be fixed! So,is there other methods to fix the problem.Thanks a lot!

Perhaps you need to change the permission to 'Anyone can access with link'.

The link is already in 'anyone who knows the link' status and I am facing the same problem

@TinaJin0228 maybe you can try gdown --no-check-certificate.

@wkentaro Hey, I am also facing exact same problem. I have tried gdown --no-check-certificate as well, and issue still persists. Permission is Anyone can access with link


feature/xpath injection

Feature for 2 XPATH injection vulnerabilities - Present both in BE and FE.

Please update readme to include the description of the added vulnerabilities


Add shape asserts across code base (instead of just relying on shape comments)

TensorFlow now has tf.debugging.assert_shapes (thanks to GPflow user @bodin-e!), which makes it straightforward to add shape checks for input and output tensors. This brings several advantages:

it helps in uncovering bugs in our code and gives us more confidence our code is behaving as intended,
it makes clear in code what shapes our code expects / returns (without relying on shape comments that may easily get out of date with the code itself),
it gives more helpful error messages to users who supply wrongly-shaped tensors as inputs.

PR #1219 started adding assert_shapes to parts of the code base, but many are still missing. The following modules have relevant tensorflow code that would probably benefit from a bunch of shape assertions.

[ ] conditionals

[ ] conditionals - not needed?
[ ] mo_conditionals -> #1219 (not 100%)
[ ] mo_sample_conditionals
[ ] sample_conditionals
[ ] uncertain_conditionals
[x] util -> #1219

[ ] covariances

...

[ ] expectations

...

[ ] kernels

...

[x] kullback_leiblers -> #1219
[ ] likelihoods
[x] logdensities (only multivariate_normal) -> #1219
[ ] mean_functions
[ ] models

...

[ ] optimizers/natgrad
[ ] quadrature
[ ] utilities/ops

If you would be willing to help with some of these, please go ahead and open PRs! :)

Good first issue? ;)

Well, I'm assuming people using GPflow and interested in contributing may have some familiarity with arrays and shapes - the shape comments tend to be there already, and there's plenty of examples of how to do it in #1219, so it's just extrapolation from there...

i don't think we should be including asserts on return values. that kind of thing should be checked with tests not runtime asserts. I'm also dubious about expecting users to ever have to look at the code to find out what a function does

should we be including asserts on return values? I'd have thought that kind of thing should be checked with tests not runtime asserts.
I'm also wary of expecting users to ever have to look at a function's body to find out what it does or requires. For example, I would expect the docs to include the type signature and docstring, but not the source code, and I would expect that to be sufficient to know how to use functionality

It'd of course be good to have a bunch of tests as well, but I don't think tests and shape asserts are mutually exclusive - it's hard to have full coverage by tests (though of course we should strive for that), and shape asserts are simply an additional set of checks. Yes, we should include shapes in the docstrings as well, but they don't always reflect the code (I've fixed some in #1219), and having shape asserts in code will make it easier to spot when there are any discrepancies.

Agree with @joelberkeley, "debugging" asserts meaningful for inputs. When one function calls another, this is awkward to have double asserts on the same value. Debugging asserts are sort of filtering preconditioners, they set what input tensors you can pass. Whereas asserts in a body are runtime checks that a function works as expected - that is actually the purpose of tests.

until I'm added as contributor, here's a note saying: add shape checks for GPR. In particular,

Could you make an issue for GPflow and emphasize that verification should be performed on early stages?

see https://stackoverflow.com/q/60061818/5986907 for more info

here's an (SO answer)[https://stackoverflow.com/a/1957656/5986907] on this topic. They explain how checking paramters should be done with standard exceptions e.g. tf.debugging.assert_shape (which looks to me like a misnomer cos a) it doesn't throw AssertionError and b) I can't see how you can turn it off). Checking the shape of a return value, if we want to do that, should be done with an assert.

Hi bodin-e,
Calling it a misnomer was mostly to highlight that assert_shapes shouldn't be used in the same places one would use assert.
As a check for function args, agreed, it's good. In other places, I agree it provides nice (relatively) dynamic documentation, but there are a few significant concerns I have:

If assert_shapes is used to check code consistency, then

users can't differentiate between a ValueError that can be caught (e.g. if args were incorrect), and one that shouldn't (e.g. because there's a mistake in the GPflow code). As I understand, one should never catch an AssertionError.
I can see it becoming a temptation to leave out tests in favour of waiting for users to inform us of errors via bug reports.

Hi bodin-e,
Calling it a misnomer was mostly to highlight that assert_shapes shouldn't be used in the same places one would use assert.
As a check for function args, agreed, it's good. In other places, I agree it provides nice (relatively) dynamic documentation, but there are a few significant concerns I have. If assert_shapes is used to check code consistency, then

users can't differentiate between a ValueError that can be caught (e.g. if args were incorrect), and one that shouldn't (e.g. because there's a mistake in the GPflow code). As I understand, one should never catch an AssertionError.
I can see it becoming a temptation to leave out tests in favour of waiting for users to inform us of errors via bug reports.

Given the first point, I don't think they should be used to check code consistency

I see. I don't agree that errors due to passing incorrect tensors to a function is to be caught by the caller (as in the meaning of catching an exception in runtime), but rather should lead to a crash (as early as possible and ideally clearly showing what is wrong) - but we might be using the word 'caught' differently here.
I do take your point that in the (highly hypothetical obviously ;-)) scenario a bug is introduced in the gpflow code, then causing non-fixable errors by the user (without gpflow source code modification), that is a problem. On the other hand, if the gpflow code bug (causing the shape error) is there - is it better it causes such an error, rather than give the user incorrect output (which might not even be discovered) or alternatively a more mystical error somewhere else further down the line?
With that said, the gpflow bug could be incorrect specification of the shape expectation (an incorrect assert_shapes shape statement) - this I agree is important to be cautious of. Unit tests are of course important to have, regardless of using assert_shapes or not. A good thing is that incorrect assert_shapes statements are very likely to fail unit tests of virtually any functionality leading to calls of such incorrect assert_shapes statements.

I would think about it like this:

I would change my mindset and treat assertions both for inputs and outputs as "executable documentation". I usually "Go to declarations" from Pycharm to look at the source code to understand how to use the code. Assertions there, together with comments help me understand the code. This also require us to write assertions with a documentation mindset -- they should make sense to the user. For example, instead of assert_shape(result, [B, T]), we should write assert_shape(result, [batch_size, time_size]).

The fact that in Tensorflow, shape assertions are defined in the debugging module is wrong. Shape assertions can be use for debugging, but that should not be their only purpose. They can be powerful documentation tool at the same time.

Shape assertions fail loudly, unlike text comments that go out of date silently.

We should not require a user to read the tests to understand what's the shape of the output.

Writing shape assertions for both input and outputs forces you (the author of the code) to think about correctness more. This process itself helps you deliver better code. Think this way: a test case checks if the code is correct for ONLY THE TEST INPUTS. Shape assertions check if the code is correct for ANY input.

Shape assertions for both inputs and outputs already helped us detect many bugs, which are not covered by any tests -- in fact, it is not easy to come up with manual tests to test all those shape situations. With assertions in place, you don't need to write any manual tests anymore, just pump in random inputs, and  if the code runs, then you are ok, otherwise, the shape assertion tells you where went wrong.


How can I use two versions at the same time on the same computer?

How can I use the two version in the same computer?
I think some old version is good,can you help me?

You can either:

Run one or more versions from source
Use pyinstaller to create a standalone executable

If you're not a Python programmer / Already familiar with Python packaging, don't do this, it's not worth your ROI. Instead use one version at a time.


scrollbar not displaying in output area

Prerequisites

[x] Have you checked the solutions in our troubleshooting guide for common problems ?
[x] Have you checked that your issue isn't already filed ?

Description
Scrollbar is not displaying in the first section of the output area (the one with clock icon). Mouse wheel also does nothing, and there is no way to scroll to the bottom of the output area.
Steps to Reproduce
for i in range(1, 200):
    print(f'{i} test')

Versions

OS version:
Windows 10

atom --version

 Atom    : 1.48.0
Electron: 5.0.13
Chrome  : 73.0.3683.121
Node    : 12.0.0

apm --version

apm  2.5.0
npm  6.14.5
node 10.20.1 x64
atom 1.48.0
python 3.7.3
git 2.17.1.windows.2
visual studio

Hydrogen version:

2.14.1
Plugins
Have you installed and activated any of the Hydrogen plugins below ?

[ ] hydrogen-python
[ ] Hydrogen Launcher
[ ] Data Explorer

Expected behavior
A scrollbar should appear on the right hand side.
Screenshots
The first section of output area does not have scrollbar

The second one does

Additional context
More people seem to be having the same problem:
https://discuss.atom.io/t/hydrogen-scrollbar-output-not-displaying/75785

This is a pretty big issue. Anyone else having it..?

@juliuscrn Can you reproduce this issue with other times out outputs (images and charts)?

@captainsafia yes, the scroll bar is also missing when I try to render a large chart:

Same issue for me for a few weeks now

I looked at this and just opened #1995, take a look if you can @captainsafia or @aviatesk


WIP: Generate global functions

Part of https://github.com/gtk-rs/gir/issues/295
Apply https://github.com/gtk-rs/gir/pull/371

@GuillaumeGomez Please look at this.
You accept use "namespace.*" as name of object for global functions or better use some other name?

Yes, this syntax is fine by me. Don't forget to document it please. :)

Ok.

@GuillaumeGomez In this and next related PR's I don't added new reexports and replaced some previously manual function.

Thanks!


Ability to override redis image in the redis deployment

Describe the feature

Currently, redis version is immutable in the Connaisseur-redis deployment. It should be editable via values file.
Optional: Is your feature request related to a problem? Please describe.

If user already has a redis image in our registries, we can't specify our registry url in the redis image.
If user wants to use a different redis image tag (Eg. enterprise redis), it is not possible.
We have these options available with the Connaisseur image already.

Optional: Implementation ideas

Optional: Additional context

Hardcoding of redis image - https://github.com/sse-secure-systems/connaisseur/blob/v3.6.1/charts/connaisseur/templates/redis.yaml#L53-L54

@phbelitz This is my first PR for this repo. Can you please review? Please let me know if anything is missing


Update build / README / CI to use west submanifest

We should update build / README / CI to refer to the Zephyr Getting Started Guide and the very simple submanifests/99-thrift.yaml, shown below
manifest:
  defaults:
    remote: upstream

  remotes:
    - name: upstream
      url-base: https://github.com/zephyrproject-rtos
    - name: cfriedt
      url-base: https://github.com/cfriedt

  projects:
    - name: gsoc-2022-thrift
      path: modules/lib/thrift
      revision: main
    # This is actually a git submodule of gsoc-2022-thrift
    - name: thrift
      remote: cfriedt
      path: modules/lib/thrift/.upstream
      # corresponds to revision: v0.16.0, but the tag does not exist in cfriedt/thrift
      revision: 63cad2ee536ead54e6ac75fb5401beb9ab2bbee2

Setup instructions only really need to say"follow the getting started guide, but before running west update, create the file submanifests/99-thrift.yaml shown below"
We don't even need our own zephyr-env.sh in that case, nor do we need our own west.yml (although it may be handy to have).

Can you also please remove the Contributors and License sections in the README? They are listed on the side of the web page already.
I'm not sure if the Qemu setup is necessary for running the basic test, so it may be better as part of the sample documentation.
The Build & Run the Testsuite section probably doesn't need to be there.


Unable to Install iOS App (we got very close!)

Hello, everyone!
I built the mediapipe iOS example using bazel through the command line, and it built successfully. I created an IPA file, but once I tried to add it to the device simulator, I got an error "Unable to install EdgeDetectionGpu App". Can anyone help?

Which MediaPipe revision are you using?

We're using the latest one on github, but made some changes to it.

Can you check you followed the steps for adding the provisioning profile? https://google.github.io/mediapipe/getting_started/building_examples.html#set-up-provisioning

Yes, we did follow those steps!

@js-0528 For more detailed instruction check out the detailed step by step blog post by one of our japanese user

Hello! I have read that! Thank you! However, I don’t think it deals with my
specific error?
On Thu, Aug 13, 2020 at 6:59 PM Ming Yong<EMAIL_ADDRESS>wrote:

@js-0528 https://github.com/js-0528 For more detailed instruction check
out the detailed step by step blog post
https://qiita.com/ponte1010/items/13249df5dab8fadea88f by one of our
japanese user
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/google/mediapipe/issues/999#issuecomment-673750545,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AM4BE2TB3T5OVFX27W4SG43SARV65ANCNFSM4P44MGFA
.

Hi @js-0528, Could you please check the above with the latest release and let us know if you see any similar observations. Thanks!


Multiple states

It would be nice to have more that 2 states for the tasks.
Currently, we have "TO-DO" and "Completed".
For example, today I found that I needed something like "Discarted" to indicate that the task was supposed to be done, but it has been discarted. We don't delete it because we would like to keep a track of what we initially planned.

Makes sense, maybe it would make sense to allow "Archiving" tasks (to keep common wording).
I'll look into this, thanks for the suggestion!

I have just realized that we also would need a "Doing" state.
Right now I am reviewing my project tasks and I don't know how many of them are already pending and which ones are already started...

Makes sense, I'm not sure about the final shape of this (in UX terms), but there's a rough aproximation deployed at the main server. (This corresponds to the branch multiple-state-options ). I'd be interested on hearing your opinion.

I'm pretty happy with the current result, merging it to master as it won't hurt.
Opinions are still welcome, I'll take another look at the possibilities when restyling the step cards.

It looks much better now! I like it!
The selection of colours is great, since it is not too much distracting

Btw, should you consider the dependencies regarding the states?
For example: A -> B
Where B depends on A completion. Then, if A is "Work in progress", and we change B to "Done", I think that at least a warning should pop up to let us know that it's weird that we finish a task whose dependency is not yet solved.


[Docs] Service Meshes update

Current State
Octarine project is shutdown.
Desired State
Remove Octarine Service Mesh from the table. here: https://docs.meshery.io/service-meshes

Contributor Guides and Resources

📚 Instructions for contributing to documentation

Meshery documentation site and source

🛠 Meshery Build & Release Strategy
🎨 Wireframes and designs for Meshery UI in Figma
🙋🏾🙋🏼 Questions: Layer5 Discussion Forum and Layer5 Community Slack

Hello @leecalcote , @Chadha93 , I would like to work on this issue and would you please elaborate the issue? Thanks


--wait does not return on editor tab close

VSCode Version: 1.18.1
OS Version: Windows 10 version 1703 (OS Build 15063.726)

Steps to Reproduce:

Open Cmd
Run git config --global core.editor "code --wait"
Open VS Code
Run git config --global -e (Opens editor tab in existing VS Code Window)
Close .gitconfig Tab, with or without saving, but do not close VS Code Window.

Expected Result:
Control should be returned at the command line after Editor is closed.
Actual Result:
Command line is still waiting for the entire VS Code window to close.
Reproduces without extensions: Yes
Notes:
I receive the expected results on another system with the VSCode and OS Version below:

VSCode Version: 1.18.1
OS Version: Windows 7 version 6.1 (Build 7601: Service Pack 1)

@Aex I cannot reproduce and it looks like you can also not reproduce on another system. What are your settings looking like? And can you attach the output of running code --verbose from the command line like so:

git config --global core.editor "code --wait --verbose"
git config --global -e

[main 8:07:22 AM] Sending env to running instance...
[main 8:07:22 AM] Sending some foreground love to the running instance: 4876
This is the point at which I close the editor tab, but leave the VS Code window open (due to other tabs). When I close VS Code completely down, I receive
[main 8:09:04 AM] Sent env to running instance. Terminating...
Settings
{
    "editor.fontFamily": "Fira Code, Consolas, 'Courier New', monospace",
    "editor.fontLigatures": true,
    "editor.rulers": [
        100
    ],
    // "editor.wordWrapColumn": 120,
    // "editor.wordWrap": "bounded",
    "editor.cursorBlinking": "phase",
    "editor.cursorStyle": "line",
    "editor.renderWhitespace": "boundary",
    "editor.renderControlCharacters": false,
    "editor.renderLineHighlight": "line",
    "editor.formatOnSave": true,
    "editor.minimap.enabled": false,
    "terminal.integrated.shell.windows": "C:\\Windows\\System32\\cmd.exe",
    "terminal.integrated.shellArgs.windows": [
        "/K",
        "C:\\cmder\\vscode.bat"
    ],
    "terminal.integrated.fontFamily": "Fira Code, Consolas, 'Courier New', monospace",
    "terminal.integrated.fontSize": 14,
    "terminal.integrated.cursorStyle": "line",
    "terminal.integrated.cursorBlinking": true,
    "workbench.iconTheme": "vscode-icons",
    "files.associations": {
        ".env": "ini",
        ".env.example": "ini",
        ".docker.env": "ini"
    },
    "extensions.ignoreRecommendations": false,
    "prettier.eslintIntegration": true,
    "prettier.tabWidth": 4,
    "workbench.colorTheme": "Monokai Dimmed",
    "gitlens.codeLens.locations": [
        "document",
        "containers",
        "blocks"
    ],
    "gitlens.gitExplorer.enabled": false,
    "prettier.printWidth": 100,
    "window.menuBarVisibility": "default",
    "emmet.includeLanguages": {},
    "gitlens.advanced.messages": {
        "suppressCommitHasNoPreviousCommitWarning": false,
        "suppressCommitNotFoundWarning": false,
        "suppressFileNotUnderSourceControlWarning": false,
        "suppressGitVersionWarning": false,
        "suppressLineUncommittedWarning": false,
        "suppressNoRepositoryWarning": false,
        "suppressUpdateNotice": false,
        "suppressWelcomeNotice": true
    },
    "explorer.confirmDragAndDrop": false,
    "window.zoomLevel": 0
}

Can you try to reproduce with our nightly insider builds? You can give our preview releases a try from: https://code.visualstudio.com/insiders/
Make sure to set code-insiders as editor for git.

I changed the editor value to code-insiders -w and I received the expected results. It appears to work correctly with the insiders build.

Ok so it must be fixed meanwhile.


javadoc.io down again

Server Down

Anything we can do to help out with stability of the side?
Javadoc generation fails due to reference of javadoc.io if it's down

It is up now. Maybe we need to find the root cause behind why it is occasionally going down.

It’s down again. Sorry the fix probably didn’t work

It’s down again. Sorry the fix probably didn’t work

yea I was waiting for weekend to deploy because weekdays' traffic is usually 10x bigger. Unfortunately the issue triggered again before deploy. Now the fix is in.


Made changes to readme, to template and to config to make compatible …

…with new versions of Kibana,Logstash and Elasticsearch

@timogoosen thanks for the PR!! Will take a look and merge in a day or too.


Dashboard: Time picker renders off screen

Time picker renders off screen, when the filter box is on the right side of a dashboard.
Expected results
It should render to the left side of the box, so that's it's fully on screen.
Actual results
Off screen.
Screenshots

How to reproduce the bug

Go to dashboard with filter box with time picker on the right side of the dashbard
Click on 'date picker'
See it renders off screen

Environment
(please complete the following information):

superset version: master

Checklist
Make sure these boxes are checked before submitting your issue - thank you!

[x] I have checked the superset logs for python stacktraces and included it here as text if there are any.
[x] I have reproduced the issue with at least the latest released version of superset.
[x] I have checked the issue tracker for the same issue and I haven't found one similar.

Similar to https://github.com/apache/incubator-superset/issues/11094 (these are the only two cosmetic / overflow issues I am aware of).
cc @junlincc

Just verified, this looks fixed, thank you!


Allow overriding default NATS configuration options

This PR allows the invoker to override the default NATS configuration options. It also bumps the version to 0.3.6.
This will help us to prevent the issue documented at https://github.com/cloudfoundry/cloud_controller_ng/issues/726.

Looks good to us!
@simonleung8 and @jberkhahn
CAPI Team


[Feature Request] Allow setting remote related WI relations

Is your feature request related to a problem? Please describe.
az boards work-item relation add has a hard requirement on the added relation being a link to a local work item that can be looked up by id. This means that it's impossible to add a relation to a remote related work item, since this does not have a local work item id that can be looked up.
Describe the solution you'd like
It should be possible to add a remote related work item relation. This would mean skipping the required id lookup, which  means adding a new command:
az boards work-item relation add-url

Additional context

Proposed solution
Instead of adding a new command, we can add a new parameter --target-url.
Cli will conditionally use between target-url and target-id

Proposed solution Instead of adding a new command, we can add a new parameter --target-url. Cli will conditionally use between target-url and target-id

@asmundg Shall I go ahead with this approach?


[Enhancement] Placeholder for Account

Is your feature request related to a problem? Please describe.
Placeholder for accounts
Describe the solution you'd like
Placeholder for accounts that will display
level, class and group per account
for ex.
"%sapi_<account_id>level%"
"%sapi<account_id>maingroup%"
"%sapi<account_id>_class%"

Also see: #400


Feat: Formatting in table view

Description

Added numeric formatting support for table-view using functions from react-number-format exported by the designsystem. See: https://github.com/Altinn/altinn-design-system/pull/165
Table cells are now aligned according to formatting.align, formatted numbers are right-aligned by default.

Currently, there is no alignment when the mobileView of the table is used, this did not look very good.

Date values from the DatePicker component are now formatted the same way as in the DatePicker itself.
As a side-effect of these changes, this also fixes the formatting of the related summary components.

Table (desktop):

Table (mobile):

Summary:

Related Issue(s)

#190
https://github.com/Altinn/altinn-design-system/pull/165

Verification

[x] Your code builds clean without any errors or warnings
[x] Manual testing done (required)
Relevant automated test added (if you find this hard, leave it and we'll help out)
[x] All tests run green

Documentation

User documentation is updated with a separate linked PR in altinn-studio-docs. (if applicable)

😍


Remove default values from basic initialiser

What does this change?
This PR removes the default values from the basic initialiser function. This is important so that as properties are added to the models, they are not missed when directly initialising the object.
Although this would mean the initialisation function will need to be more verbose, it will ensure properties are all correctly initialised.
You can see an example in the iOS live repo where this caused problems - https://github.com/guardian/ios-live/pull/8207.

How to test
I have tested this using the example project in the repo - you can see now that the AddressBook initialiser requires the value isCurrent to be set.

How can we measure success?

Have we considered potential risks?

Images

Accessibility

[ ] Tested with screen reader
[ ] Navigable with keyboard
[ ] Colour contrast passed
[ ] The change doesn't use only colour to convey meaning

Looks good. Lucky, we had already separated the default value into its own function. Should we delete that function too?

I had thought of doing that but it's actually still used as part of the proto init to provide a default value for optional fields here https://github.com/guardian/swift-buffet/blob/3e5c417325ec51b5d9582bbf7f14abb66ac3ee0d/Sources/SwiftBuffet/Generator.swift#L302


Build fails since Docker daemon is not running.

First build task to invoke a Docker command (docker system prune) failed likely because the daemon is not running. For example, (https://devdiv.visualstudio.com/DefaultCollection/DevDiv/_build/index?buildId=1052676&_a=summary)
2017-10-11T03:01:05.4286013Z ##[command]docker system prune -a -f
2017-10-11T03:01:05.5001821Z error during connect: Post http://%2F%2F.%2Fpipe%2Fdocker_engine/v1.32/containers/prune: open //./pipe/docker_engine: The system cannot find the file specified. In the default daemon configuration on Windows, the docker client must be run elevated to connect. This error may also indicate that the docker daemon is not running.
2017-10-11T03:01:05.5470492Z ##[error]Process completed with exit code 1.

Possible solutions -

Ensure the Start Docker when you log in option is enabled.
Restart Docker service as the first step in the build. Start-Service -Name com.docker.service and  Start-Service -Name docker

I think fixing this should be a priority.  These failure are consistent and always happen at the most inconvenient times.  I would suggest pinging Ilya on your proposed fix.

Requested DDIT to set Recovery options on Docker services for each VM in DotNet-Infra pool. Refer http://ddweb/techease/TicketEdit.aspx?id=1536152

Though the service Recovery options are set correctly, it is required to ensure C:\Program Files\Docker\Docker\Docker for Windows.exe is included in the startup programs.

We can consider starting Docker for Windows.exe in a build task through the script shown below -
$retryCount = 0
$retries = 5

while ($retryCount -lt $retries)
{
    Get-Process -Name "Docker for Windows" -ErrorVariable dockerProc
    if ($dockerProc -ne 0)
    {
        $dockerExe = Join-Path ${$env:ProgramFiles} "Docker\Docker\Docker for Windows.exe"
        if (-not (Test-Path $dockerExe -PathType Leaf))
        {
            $dockerExe = Join-Path ${env:ProgramFiles(x86)} "Docker\Docker\Docker for Windows.exe"
        }
        & "$dockerExe"
        sleep -Seconds (Get-Random -Minimum 30 -Maximum 60)
        $retryCount += 1
    }
    else
    {
        break
    }
}

This issue has been resolved by DDIT.

Thanks Michael for working with Ilya in resolving this issue.


feat(client): add -d to gofmt to get more info on style failures

Does adding -d (show diffs instead of rewrite files) actually change the gofmt output on failure? I think -l (just list the files) effectively overrides it. That is, I got the same output from make -C client/ test-style before and after this change, whether or not the go code has an error that gofmt notices.

it works but blagh... our makefile isn't doing us any favours:
make test
/bin/sh: -c: line 0: syntax error near unexpected token `('
/bin/sh: -c: line 0: `if [ cmd/ps.go diff cmd/ps.go gofmt/cmd/ps.go --- /var/folders/cb/xyx1dpd527z951dzq3f80r7r0000gp/T/gofmt219126809	2016-02-17 12:45:53.000000000 -0800 +++ /var/folders/cb/xyx1dpd527z951dzq3f80r7r0000gp/T/gofmt557976740	2016-02-17 12:45:53.000000000 -0800 @@ -118,7 +118,7 @@    	if len(processes) == 0 {  		fmt.Println("Could not find any processes to restart") -	} else{ +	} else {  		fmt.Printf("done in %ds\n", int(time.Since(startTime).Seconds()))  		printProcesses(appID, processes)  	} ]; then 	echo "gofmt check failed:"; gofmt -e -l -s -d deis.go deis_test.go cmd controller/api controller/client controller/models/apps controller/models/auth controller/models/builds controller/models/certs controller/models/config controller/models/domains controller/models/keys controller/models/perms controller/models/ps controller/models/releases controller/models/users parser pkg/git pkg/webbrowser; exit 1; fi'


Override Realm file by another Realm file has same structure

Goals
Complete replace all objects from one Realm to other Realm.
At this post, i'm try to achieve that by delete all files of the RealmA need to be replaced, then copy all files of the RealmB will overriding to that path.
Expected Results
RealmA should have same data as RealmB
Actual Results
RealmA keeps its old data, RealmB can't override its data to RealmA
Code Sample
class Database {
    private let identifyName: String

    private lazy var realmConfiguration: Realm.Configuration = {
        var config = Realm.Configuration()
        config.fileURL = config.fileURL!.deletingLastPathComponent().appendingPathComponent("\(self.identifyName).realm")
        return config
    }()

    private var realm: Realm {
        return try! Realm(configuration: self.realmConfiguration)
    }

    init(indentifyName: String) {
        self.identifyName = indentifyName
    }

    //MARK: add
    func addPet(withName name: String) {
        let obj = Pet()
        obj.name = name
        try! realm.write {
            self.realm.add(obj)
        }
    }

    //MARK: complete delete Realm file
    func deleteRealmFile() {
        try! realm.write {
            if let path = self.realmConfiguration.fileURL?.path {
                let pathLock = path + ".Lock"
                let pathManagement = path + ".Management"
                try! FileManager.default.removeItem(atPath: path)
                try! FileManager.default.removeItem(atPath: pathLock)
                try! FileManager.default.removeItem(atPath: pathManagement)
            }
        }
    }

    //MARK: copy file function
    func overrideByWriteCopy(toManager managerDataNeedToBeOverrided: Database) {
        let otherUrl = managerDataNeedToBeOverrided.realm.configuration.fileURL!
        managerDataNeedToBeOverrided.deleteRealmFile()
        try! self.realm.writeCopy(toFile: otherUrl)
    }

    func overrideByCopyFile(toManager managerDataNeedToBeOverrided: Database) {
        let thisUrl     = self.realm.configuration.fileURL!
        let otherUrl    = managerDataNeedToBeOverrided.realm.configuration.fileURL!
        managerDataNeedToBeOverrided.deleteRealmFile()
        try! FileManager.default.copyItem(at: thisUrl, to: otherUrl)
    }
}

class Pet: Object {
    dynamic var name: String = ""
}

let db1: Database = Database(indentifyName: "db1")
let db2: Database = Database(indentifyName: "db2")

// Insert code here to initialize your application
db1.addPet(withName: "1")
db2.addPet(withName: "A")
//db1: [1], db2: [A]

// db1.overrideByCopyFile(toManager: db2) // Already tried this, same result
db1.overrideByWriteCopy(toManager: db2)
//Result: db1: [1], db2: [A,2]
//Expected: db1: [1], db2: [1,2]

Sample project here:
RealmUnicode.zip
Version of Realm and Tooling
Realm version: "v2.1.2"
Xcode version: Version 8.1 (8B62)
Swift version: 3.0.1
OSX: 10.11.6 (15G1108)

Realm Swift doesn't support deleting a Realm's files after it's been opened.
Your options are to:

Delete the original Realm files without opening the Realm, copy the new Realm files to the original location, and then open the Realm at then open the Realm at the original location.
Open both Realm files, and then use the Realm APIs to delete objects from the first Realm and copy them from the second Realm to the first Realm.

You might want to investigate the myRealm.deleteAll() API to delete all objects from a Realm.

Is there anyway to safely close the Realm while it has been opened (to delete it) ?
My data is about thousands to millions and it need to be replaced completely each time update, so i rather a solution that can achieve it easily, not by taking some times to read all data to update.
I've used realm.deleteAll() then copy all (func overrideByDeleteAll bellow)
Currently i'm using override data by compare two realm to insert/delete needed objects (func overrideByCompare bellow) (because my new data not use to completely different from old data)
func enumerateAllItemThat(notBeingContainedByManager otherManager: HCDItemDataManager, andExecuteAction action: (HCDItemObject)->Void) {
    self.allItems.forEach {
        if otherManager.allItems.filterByPath(exactly: $0.path).isEmpty {
            action($0)
        }
    }
}
//MARK: - Override By delete and copy file
func overrideByCompare(toManager managerDataNeedToBeOverrided: HCDItemDataManager) {
//copy needed files
    self.writeWithBlock {
        self.enumerateAllItemThat(notBeingContainedByManager: managerDataNeedToBeOverrided) {
            managerDataNeedToBeOverrided.add(fromItem: $0.item)
        }
    }
//delete needed files
    managerDataNeedToBeOverrided.writeWithBlock {
        managerDataNeedToBeOverrided.enumerateAllItemThat(notBeingContainedByManager: self) {
            managerDataNeedToBeOverrided.realm.delete($0)
        }
    }
}
//MARK: - Override By delete all then copy all
func overrideByDeleteAll(toManager managerDataNeedToBeOverrided: HCDItemDataManager) {
    managerDataNeedToBeOverrided.deleteAll()
    self.allItems.forEach {
        managerDataNeedToBeOverrided.add(fromItem: $0.item)
    }
}

If there're any better solutions or any ways to improve my code, please let me know.
Thank you guys a lot.

Hi, unfortunately Realm Cocoa does not support closing a Realm after it's been opened. Your options are to either copy the files before opening either Realm, or to use Realm's object APIs to delete and copy objects once the Realms are opened. Thanks for getting in touch with us!


Cuda OutOfMemory error for DPO Finetuning

Hi,
I'm working on finetuning Zephyr7b with my own dataset with the DPO form-factor :
{
   "prompt":[]str,
   "chosen":[]str,
   "rejected":[]str,
}

I have used the code you provide in your examples : dpo.py and i ran it on an H100 instance with 80Gigs of VRAM, but i have this error :
(venv) user@instance-h100-1:~/test/FT$ python dpo_finetuning.py --max_length=4096
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  6.27it/s]
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:01<00:00,  6.41it/s]
/home/user/test/venv/lib/python3.10/site-packages/trl/trainer/dpo_trainer.py:176: UserWarning: When using DPODataCollatorWithPadding, you should set `max_prompt_length` in the DPOTrainer's init it will be set to `128` by default, but you should do it yourself in the future.
  warnings.warn(
[codecarbon INFO @ 15:57:46] [setup] RAM Tracking...
[codecarbon INFO @ 15:57:46] [setup] GPU Tracking...
[codecarbon INFO @ 15:57:46] Tracking Nvidia GPU via pynvml
[codecarbon INFO @ 15:57:46] [setup] CPU Tracking...
[codecarbon WARNING @ 15:57:46] No CPU tracking mode found. Falling back on CPU constant mode.
[codecarbon WARNING @ 15:57:47] We saw that you have a AMD EPYC 9334 32-Core Processor but we don't know it. Please contact us.
[codecarbon INFO @ 15:57:47] CPU Model on constant consumption mode: AMD EPYC 9334 32-Core Processor
[codecarbon INFO @ 15:57:47] >>> Tracker's metadata:
[codecarbon INFO @ 15:57:47]   Platform system: Linux-5.15.0-89-generic-x86_64-with-glibc2.35
[codecarbon INFO @ 15:57:47]   Python version: 3.10.12
[codecarbon INFO @ 15:57:47]   CodeCarbon version: 2.2.3
[codecarbon INFO @ 15:57:47]   Available RAM : 235.941 GB
[codecarbon INFO @ 15:57:47]   CPU count: 24
[codecarbon INFO @ 15:57:47]   CPU model: AMD EPYC 9334 32-Core Processor
[codecarbon INFO @ 15:57:47]   GPU count: 1
[codecarbon INFO @ 15:57:47]   GPU model: 1 x NVIDIA H100 PCIe
wandb: (1) Create a W&B account
wandb: (2) Use an existing W&B account
wandb: (3) Don't visualize my results
wandb: Enter your choice: 3
wandb: You chose "Don't visualize my results"
wandb: Tracking run with wandb version 0.16.0
wandb: W&B syncing is set to `offline` in this directory.
wandb: Run `wandb online` or set WANDB_MODE=online to enable cloud syncing.
  0%|                                                                                | 0/1000 [00:00<?, ?it/s]Traceback (most recent call last):
  File "/home/user/test/FT/dpo_finetuning.py", line 163, in <module>
    dpo_trainer.train()
  File "/home/user/test/venv/lib/python3.10/site-packages/transformers/trainer.py", line 1591, in train
    return inner_training_loop(
  File "/home/user/test/venv/lib/python3.10/site-packages/transformers/trainer.py", line 1892, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/home/user/test/venv/lib/python3.10/site-packages/transformers/trainer.py", line 2776, in training_step
    loss = self.compute_loss(model, inputs)
  File "/home/user/test/venv/lib/python3.10/site-packages/trl/trainer/dpo_trainer.py", line 500, in compute_loss
    loss, metrics = self.get_batch_metrics(model, inputs, train_eval="train")
  File "/home/user/test/venv/lib/python3.10/site-packages/trl/trainer/dpo_trainer.py", line 451, in get_batch_metrics
    ) = self.concatenated_forward(model, batch)
  File "/home/user/test/venv/lib/python3.10/site-packages/trl/trainer/dpo_trainer.py", line 417, in concatenated_forward
    all_logits = model(
  File "/home/user/test/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/test/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/test/venv/lib/python3.10/site-packages/accelerate/utils/operations.py", line 659, in forward
    return model_forward(*args, **kwargs)
  File "/home/user/test/venv/lib/python3.10/site-packages/accelerate/utils/operations.py", line 647, in __call__
    return convert_to_fp32(self.model_forward(*args, **kwargs))
  File "/home/user/test/venv/lib/python3.10/site-packages/torch/amp/autocast_mode.py", line 16, in decorate_autocast
    return func(*args, **kwargs)
  File "/home/user/test/venv/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 1045, in forward
    outputs = self.model(
  File "/home/user/test/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/test/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/test/venv/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 932, in forward
    layer_outputs = decoder_layer(
  File "/home/user/test/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/test/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/test/venv/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 621, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
  File "/home/user/test/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/test/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/test/venv/lib/python3.10/site-packages/transformers/models/mistral/modeling_mistral.py", line 262, in forward
    key_states = self.k_proj(hidden_states)
  File "/home/user/test/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/user/test/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/test/venv/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB. GPU 0 has a total capacty of 79.11 GiB of which 6.62 MiB is free. Including non-PyTorch memory, this process has 79.09 GiB memory in use. Of the allocated memory 77.89 GiB is allocated by PyTorch, and 641.53 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

I've changed the batch size but it doesn't make it work.
Any hint ?
Thanks a lot

Ok ! i've fixed it for now and fine-tuning is running :
if you have the same issue, don't forget to add some args while loading the model like :
low_cpu_mem_usage=True, torch_dtype=torch.float16it should fix the memory error.
Also be careful how you're loading your dataset !
If you have any questions, let me know

Hi,
Unfortunately, the DPO Finetuning didn't go well. After many tries at the 50th percent, it's throwing this error :

Traceback (most recent call last):
  File "/home/mathis/ticketing-data-poc/FT/dpo_finetuning.py", line 159, in <module>
  File "/home/mathis/ticketing-data-poc/venv/lib/python3.10/site-packages/transformers/trainer.py", line 1556, in train
    return inner_training_loop(
  File "/home/mathis/ticketing-data-poc/venv/lib/python3.10/site-packages/transformers/trainer.py", line 1934, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/home/mathis/ticketing-data-poc/venv/lib/python3.10/site-packages/transformers/trainer.py", line 2283, in _maybe_log_save_evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
  File "/home/mathis/ticketing-data-poc/venv/lib/python3.10/site-packages/transformers/trainer.py", line 3026, in evaluate
    output = eval_loop(
  File "/home/mathis/ticketing-data-poc/venv/lib/python3.10/site-packages/trl/trainer/dpo_trainer.py", line 616, in evaluation_loop
    policy_output_decoded, ref_output_decoded = self.get_batch_samples(self.model, random_batch)
  File "/home/mathis/ticketing-data-poc/venv/lib/python3.10/site-packages/trl/trainer/dpo_trainer.py", line 513, in get_batch_samples
    policy_output = model.generate(
  File "/home/mathis/ticketing-data-poc/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/mathis/ticketing-data-poc/venv/lib/python3.10/site-packages/transformers/generation/utils.py", line 1800, in generate
    return self.sample(
  File "/home/mathis/ticketing-data-poc/venv/lib/python3.10/site-packages/transformers/generation/utils.py", line 2933, in sample
    next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
RuntimeError: probability tensor contains either `inf`, `nan` or element < 0

I tried to decrease Learning Rate but it doesn't change ...
Any suggestion ?

Looks indeed like a divergence issue. Have you tried with the default dataset before using your own custom one?

Hi, Unfortunately, the DPO Finetuning didn't go well. After many tries at the 50th percent, it's throwing this error :
Traceback (most recent call last):
  File "/home/user/test/FT/dpo_finetuning.py", line 159, in <module>
  File "/home/user/test/venv/lib/python3.10/site-packages/transformers/trainer.py", line 1556, in train
    return inner_training_loop(
  File "/home/user/test/venv/lib/python3.10/site-packages/transformers/trainer.py", line 1934, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/home/user/test/venv/lib/python3.10/site-packages/transformers/trainer.py", line 2283, in _maybe_log_save_evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
  File "/home/user/test/venv/lib/python3.10/site-packages/transformers/trainer.py", line 3026, in evaluate
    output = eval_loop(
  File "/home/user/test/venv/lib/python3.10/site-packages/trl/trainer/dpo_trainer.py", line 616, in evaluation_loop
    policy_output_decoded, ref_output_decoded = self.get_batch_samples(self.model, random_batch)
  File "/home/user/test/venv/lib/python3.10/site-packages/trl/trainer/dpo_trainer.py", line 513, in get_batch_samples
    policy_output = model.generate(
  File "/home/user/test/venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/user/test/venv/lib/python3.10/site-packages/transformers/generation/utils.py", line 1800, in generate
    return self.sample(
  File "/home/user/test/venv/lib/python3.10/site-packages/transformers/generation/utils.py", line 2933, in sample
    next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
RuntimeError: probability tensor contains either `inf`, `nan` or element < 0

I tried to decrease Learning Rate but it doesn't change ...
Any suggestion ?

@mathis-lambert  Did you solve this? I'm getting exactly the same in iteration 500 with the default dataset.

Hi @guyadamSC
Yes, if I remember well, I managed to handle this issue but I don't remember how. I'm currently in vacation, so I might be back with an answer in some days.
Merry Christmas and Happy holidays 🎄

Sure, thanks a lot, enjoy your holiday!
On Tue, Dec 26, 2023 at 2:48 PM Mathis @.***> wrote:

Hi @guyadamSC https://github.com/guyadamSC
Yes, if I remember well, I managed to handle this issue but I don't
remember how. I'm currently in vacation, so I might be back with an answer
in some days.
Merry Christmas and Happy holidays 🎄
—
Reply to this email directly, view it on GitHub
https://github.com/huggingface/trl/issues/1024#issuecomment-1869524117,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AXAQDQXO4C4KWPWVFCZ4QNTYLLBQDAVCNFSM6AAAAAA7WPMHY6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNRZGUZDIMJRG4
.
You are receiving this because you were mentioned.Message ID:
@.***>

i have the same problem! It also threw the same error at 50%. Did you solve this?

HI @guyadamSC @AIR-hl,
I'm back to try to help you ...
Please can you explain a bit deeper what dataset and model are u using, and on what hardware ?
And of course, what's your trainer config (batch_size, gradient _accumulation ...)
Maybe i'll be able to help you with these info
Regard

hello! i meet the same problem, i guess my data is too long ,i use A100 40G and max_prompt length is 512,max_length is 1024, but i cant change the training data ,what should i do to solve the cuda out of memory error?  thanks!!!

hello! i meet the same problem, i guess my data is too long ,i use A100 40G and max_prompt length is 512,max_length is 1024, but i cant change the training data ,what should i do to solve the cuda out of memory error?  thanks!!!

Hi !
What model are u using ?

hello , my model is codellama-7b-python and i do the SFT before the DPO , and here is my dataset format:
,
now i am using the batch_size 1 and per_decive_train(eval)_batch_size  1  and  gradient_accumulation_steps 4 , it starts training and the output is like:

i dont know if   the data is normal

here is my code:
import os
from dataclasses import dataclass, field
from typing import Dict, Optional
import json
import torch
from accelerate import Accelerator
from datasets import Dataset, load_dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer, HfArgumentParser, TrainingArguments, set_seed
from trl import DPOTrainer
os.environ["WANDB_DISABLED"]="true"
class ScriptArguments():
beta = 0.1
model_name_or_path= "/public/home/zhouxiabing/data/xinj/RL_Model/model"
learning_rate= 1.4e-6
lr_scheduler_type = "cosine"
warmup_steps = 100
optimizer_type = "adamw_torch"
per_device_train_batch_size = 1
per_device_eval_batch_size =  1
batch_size=1
gradient_accumulation_steps = 4

lora_alpha = 16
lora_dropout=0.05
lora_r = 64

max_prompt_length=256
max_length= 512
max_steps=1000
logging_steps=10
save_steps= 100
eval_steps= 100

output_dir="./results"
load_best_model_at_end=True
save_total_limit=3

def get_dataset(
data_dir:str
):
dataset = load_dataset('json',data_files=data_dir, split="train")
def map_data(samples):
    instruct="Given a piece of code, output the corresponding implementation idea.\n"
    return {
        "prompt":instruct + samples['text'],
        "chosen":samples["chosen"],
        "rejected":samples['rejected']
    }
return dataset.map(map_data)

if name=="main":
# parser=HfArgumentParser(ScriptArguments)
# args=parser.parse_args_into_dataclasses()[0]
args= ScriptArguments()
#1. 加载预训练模型
model = AutoModelForCausalLM.from_pretrained(
    args.model_name_or_path,
    load_in_8bit=True,
    low_cpu_mem_usage=True,
    torch_dtype=torch.float16,
    device_map="cuda:0")
model.config.use_cache = True

tokenizer = AutoTokenizer.from_pretrained("/public/home/zhouxiabing/data/xinj/codellama-7b-python")
tokenizer.add_eos_token = True
tokenizer.pad_token_id = 0
tokenizer.padding_side = "left"

#2.加载数据集
train_dataset= get_dataset("new_train.jsonl")

# train_dataset = train_dataset.filter(
#     lambda x: len(x["prompt"]) + len(x["chosen"]) <= args.max_length
#     and len(x["prompt"]) + len(x["rejected"]) <= args.max_length
# )

eval_dataset= get_dataset("new_test.jsonl")

# eval_dataset = eval_dataset.filter(
#     lambda x: len(x["prompt"]) + len(x["chosen"]) <= args.max_length
#     and len(x["prompt"]) + len(x["rejected"]) <= args.max_length
# )

#3. 准备training args
training_args= TrainingArguments(
    per_device_eval_batch_size=args.per_device_eval_batch_size,
    per_device_train_batch_size=args.per_device_train_batch_size,
    gradient_accumulation_steps=args.gradient_accumulation_steps,
    logging_steps=args.logging_steps,
    eval_steps=100,
    save_steps=100,
    num_train_epochs= 2,
    optim="adamw_torch",
    evaluation_strategy="steps", # if val_set_size > 0 else "no",
    save_strategy="steps",
    save_total_limit=3,
    fp16=True,
    output_dir=args.output_dir,
    load_best_model_at_end=True,
    learning_rate= args.learning_rate,
    warmup_steps=50
)
config = LoraConfig(
    r=64,
    lora_alpha=16,
    target_modules=[
    "v_proj",
    "up_proj",
    "down_proj",
    "k_proj",
    "gate_proj",
    "q_proj",
    "o_proj"
],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

#4.准备DPO trainer
dpo_trainer=DPOTrainer(
    model,
    ref_model=None,
    args=training_args,
    beta=0,
    train_dataset = train_dataset,
    eval_dataset = eval_dataset,
    tokenizer = tokenizer,
    peft_config = config,
    max_prompt_length =256,
    max_length = 512
)

#5.训练
dpo_trainer.train()
dpo_trainer.save_model(args.output_dir)

output_dir=os.path.join(args.output_dir,"final_checkpoint")
# dpo_trainer.model.save_pretrained(output_dir)

here is the code
`import os
from dataclasses import dataclass, field
from typing import Dict, Optional
import json
import torch
from accelerate import Accelerator
from datasets import Dataset, load_dataset
from peft import LoraConfig
from transformers import AutoModelForCausalLM, AutoTokenizer, HfArgumentParser, TrainingArguments, set_seed
from trl import DPOTrainer
os.environ["WANDB_DISABLED"]="true"
class ScriptArguments():
beta = 0.1
model_name_or_path= "/public/home/zhouxiabing/data/xinj/RL_Model/model"
learning_rate= 1.4e-6
lr_scheduler_type = "cosine"
warmup_steps = 100
optimizer_type = "adamw_torch"
per_device_train_batch_size = 1
per_device_eval_batch_size =  1
batch_size=1
gradient_accumulation_steps = 4

lora_alpha = 16
lora_dropout=0.05
lora_r = 64

max_prompt_length=256
max_length= 512
max_steps=1000
logging_steps=10
save_steps= 100
eval_steps= 100

output_dir="./results"
load_best_model_at_end=True
save_total_limit=3

def get_dataset(
data_dir:str
):
dataset = load_dataset('json',data_files=data_dir, split="train")
def map_data(samples):
    instruct="Given a piece of code, output the corresponding implementation idea.\n"
    return {
        "prompt":instruct + samples['text'],
        "chosen":samples["chosen"],
        "rejected":samples['rejected']
    }
return dataset.map(map_data)

if name=="main":
# parser=HfArgumentParser(ScriptArguments)
# args=parser.parse_args_into_dataclasses()[0]
args= ScriptArguments()
#1. 加载预训练模型
model = AutoModelForCausalLM.from_pretrained(
    args.model_name_or_path,
    load_in_8bit=True,
    low_cpu_mem_usage=True,
    torch_dtype=torch.float16,
    device_map="cuda:0")
model.config.use_cache = True

tokenizer = AutoTokenizer.from_pretrained("/public/home/zhouxiabing/data/xinj/codellama-7b-python")
tokenizer.add_eos_token = True
tokenizer.pad_token_id = 0
tokenizer.padding_side = "left"

#2.加载数据集
train_dataset= get_dataset("new_train.jsonl")

# train_dataset = train_dataset.filter(
#     lambda x: len(x["prompt"]) + len(x["chosen"]) <= args.max_length
#     and len(x["prompt"]) + len(x["rejected"]) <= args.max_length
# )

eval_dataset= get_dataset("new_test.jsonl")

# eval_dataset = eval_dataset.filter(
#     lambda x: len(x["prompt"]) + len(x["chosen"]) <= args.max_length
#     and len(x["prompt"]) + len(x["rejected"]) <= args.max_length
# )

#3. 准备training args
training_args= TrainingArguments(
    per_device_eval_batch_size=args.per_device_eval_batch_size,
    per_device_train_batch_size=args.per_device_train_batch_size,
    gradient_accumulation_steps=args.gradient_accumulation_steps,
    logging_steps=args.logging_steps,
    eval_steps=100,
    save_steps=100,
    num_train_epochs= 2,
    optim="adamw_torch",
    evaluation_strategy="steps", # if val_set_size > 0 else "no",
    save_strategy="steps",
    save_total_limit=3,
    fp16=True,
    output_dir=args.output_dir,
    load_best_model_at_end=True,
    learning_rate= args.learning_rate,
    warmup_steps=50
)
config = LoraConfig(
    r=64,
    lora_alpha=16,
    target_modules=[
    "v_proj",
    "up_proj",
    "down_proj",
    "k_proj",
    "gate_proj",
    "q_proj",
    "o_proj"
],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

#4.准备DPO trainer
dpo_trainer=DPOTrainer(
    model,
    ref_model=None,
    args=training_args,
    beta=0,
    train_dataset = train_dataset,
    eval_dataset = eval_dataset,
    tokenizer = tokenizer,
    peft_config = config,
    max_prompt_length =256,
    max_length = 512
)

#5.训练
dpo_trainer.train()
dpo_trainer.save_model(args.output_dir)

output_dir=os.path.join(args.output_dir,"final_checkpoint")
# dpo_trainer.model.save_pretrained(output_dir)

`

@virt9 what's exactly the type of the model u're loading ?

hello ,  the model is lora safetensors after SFT ,

and the config is :
{   "alpha_pattern": {},   "auto_mapping": null,   "base_model_name_or_path": "/public/home/zhouxiabing/data/xinj/codellama-7b-python",   "bias": "none",   "fan_in_fan_out": false,   "inference_mode": true,   "init_lora_weights": true,   "layer_replication": null,   "layers_pattern": null,   "layers_to_transform": null,   "loftq_config": {},   "lora_alpha": 16,   "lora_dropout": 0.05,   "megatron_config": null,   "megatron_core": "megatron.core",   "modules_to_save": null,   "peft_type": "LORA",   "r": 64,   "rank_pattern": {},   "revision": null,   "target_modules": [     "v_proj",     "down_proj",     "q_proj",     "up_proj",     "gate_proj",     "k_proj",     "o_proj"   ],   "task_type": "CAUSAL_LM",   "use_dora": false,   "use_rslora": false }

@virt9
Okay,

Do the traceback show when the error was raised ?
What's the size (in MB) of your DS ?

@mathis-lambert  hello，now i am following the code i mentioned above to train the model  and it keeps  training with no cuda error , but the loss is nan and the rewards is nan , now i am going to solve the new problems , thanks a lot for your help !!

@mathis-lambert  hello，now i am following the code i mentioned above to train the model  and it keeps  training with no cuda error , but the loss is nan and the rewards is nan , now i am going to solve the new problems , thanks a lot for your help !!

Ok nice you don't have CUDA OUT OF MEMORY error
You can stop the training, it is failed.
I already experienced this issue, did the loss and rewards had a value till epoch 50 ?

@mathis-lambert  hello，now i am following the code i mentioned above to train the model  and it keeps  training with no cuda error , but the loss is nan and the rewards is nan , now i am going to solve the new problems , thanks a lot for your help !!

Ok nice you don't have CUDA OUT OF MEMORY error You can stop the training, it is failed. I already experienced this issue, did the loss and rewards had a value till epoch 50 ?

oh yes ,to prevent the cuda out of memory .i set all batch_size 1 ,it works for me.  so i have 1770+ steps ,and the first  hundreds steps the value is right but now it is 0 ,that's why ?

@virt9 damn I've got this same behavior but i can't remember how i solved it ...
Then, for the batch_size yup it's normal you can't go above 1.
You only have 40gigs of VRAM and a Lora fine tuning loads the weights in FP32 so it takes 4 times the parameter size in gigs :
7B model in FP32 should weigh around 28GB + Dataset size.
That's why.

if the loss is 0, maybe the fp16 is overflow, please try bf16
---Original---
From: @.>
Date: Sat, May 4, 2024 18:14 PM
To: @.>;
Cc: "Shihyueh<EMAIL_ADDRESS>Subject: Re: [huggingface/trl] Cuda OutOfMemory error for DPO Finetuning(Issue #1024)
@mathis-lambert  hello，now i am following the code i mentioned above to train the model  and it keeps  training with no cuda error , but the loss is nan and the rewards is nan , now i am going to solve the new problems , thanks a lot for your help !!
Ok nice you don't have CUDA OUT OF MEMORY error You can stop the training, it is failed. I already experienced this issue, did the loss and rewards had a value till epoch 50 ?
oh yes ,to prevent the cuda out of memory .i set all batch_size 1 ,it works for me.  so i have 1770+ steps ,and the first  hundreds steps the value is right but now it is 0 ,that's why ?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>

With this
´torch_dtype=torch.bfloat16´

@mathis-lambert  thanks a lot , i will try it later

@mathis-lambert  sorry to bother you again, i try bf16 and after 1200steps the loss is 0  again ,and i found a solution to higher my batch_size,  i make it from 1 to 2 ,but cuda out of memory error comes again. did you meet this? thanks!

sorry, i have no idea. maybe you should make an new issue again
---Original---
From: @.>
Date: Sun, May 5, 2024 11:09 AM
To: @.>;
Cc: "Shihyueh<EMAIL_ADDRESS>Subject: Re: [huggingface/trl] Cuda OutOfMemory error for DPO Finetuning(Issue #1024)
@mathis-lambert  sorry to bother you again, i try bf16 and after 1200steps the loss is 0  again ,and i found a solution to higher my batch_size,  i make it from 1 to 2 ,but cuda out of memory error comes again. did you meet this? thanks!
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you were mentioned.Message ID: @.***>

OK, thanks a lot


Ability to save model

Need functionality for saving trained TensorFlow model.

Pull request #12 solves this issue.


"FIX: use cstdint"

Summary
use cstdint
Purpose
to clarfy the size of range of integer
Contents
replace:
unsigned int -> std::uint16_t (where the integer is guaranteed that does not take the value over 3001)
unsigned char -> std::uint8_t
long long int -> std::int64_t
unsigned long long int -> std::uint64_t
Testing Methods Performed
bazel build all

9a800b6
unsigned int -> std::uint32_t (even if the integer is guranteed that does not take the value over uint8_t or uint16_t

@sakamoto-souta
cloud you rebase this pr branch to latest main branch: 1315414c5e83647bbd131ef9f56a60a66f7f71c8 ?

@arukuka
I rebased


Add travis config (and make tests runnable on emacs 23)

There was a bit more to this than I initially expected...
The rust-mode code has a few things in it that show intent to keep it compatible with emacs 23 as well as emacs 24, so the .travis.yml file here tests it against emacs 23, 24 and the latest snapshot build.  To make it successfully test on emacs 23 I had to change the test code (though not the actual rust-mode code under test) to eliminate some use of forms that were newly introduced in emacs 24.
The way it installs emacs in the Travis VM is taken from the example at https://github.com/rolandwalker/emacs-travis.  Other repos seem using the same approach as well--a very well known one is Magit.
There are a few unfortunate things about it:

The PPA that it uses to get emacs 24 now proclaims itself deprecated, although for the time being it still works.
Because it uses apt-get to install emacs, it can't use Travis's new container infrastructure.  The Travis team appears to be working on lifting this limitation, so this seems to be only a temporary problem.

An alternative approach would be to have it download an emacs source tarball right from a GNU mirror, and rebuild emacs from source each time.  That would eliminate dependency on no-longer-interested third parties, but would also complicate the configuration in this repo and might make the travis builds take a while.  (I haven't tried it, so I can't really say for sure.)
Here is a successful Travis run with this version from my own fork.

This looks pretty reasonable, and it'd be awesome to have travis running the unit tests.

@MicahChalmer what else is needed to get travis going?

All it needs now is for someone with admin access on this repo to go to travis-ci.org and flip the switch.

Done.

Looks like it works great, thanks @MicahChalmer: https://github.com/rust-lang/rust-mode/pull/24, https://travis-ci.org/rust-lang/rust-mode/builds/49740310 (tests seem to fail on emac23?).


Replace source date member fetching with new procedure

This will solve member CCSID issues with SQL mode once and for all.
@chrjorgensen and I have been going on for months about replacing our logic to fetch member aliases with our runner, with new procedure that either returns the rows or writes to a file. Sadly, returning the rows from the procedure doesn't solve our ever ending CCSID issues with source members.
For those who are reading this and wondering why we don't use CPYTOSTMF or CPYTOIMPF: neither of those commands return the source dates, which is what we need.
So, Christian wrote this procedure, which generates the alias into QTEMP and then writes the file to the IFS. The implementation is good, but inside of Code for IBM i we already create the aliases in the temporary library so we can re-use them when saving members again.
call NCCJ.Get_Member_Content( LIB => 'nccj', FIL => 'qclsrc', MBR => 'burncpu', STMF => 'abc.txt',  INC_SEQ => 'Y', INC_DATE => 'Y' )

Source
-- Get member content, optionally write to streamfile and include seq and date

set schema NCCJ;

create or replace procedure Get_Member_Content(
  lib varchar( 10 )
, fil varchar( 10 )
, mbr varchar( 10 )
, stmf varchar( 1024 ) default ''
, inc_seq char( 1 ) default 'N'
, inc_date char( 1 ) default 'N'
)
specific GETMBRCONT
program type sub
result sets 1

begin
  declare LF char(1) ccsid 1208;

  declare v_Sql varchar( 1024 );
  declare v_SourceLine varchar( 512 );
  declare v_StmfOut clob(<PHONE_NUMBER> ) ccsid 1208 default '';
  declare v_LineCount int default 0;

  declare at_end int default 0;

  declare C cursor for S;

    declare continue handler for SQLSTATE '02000'
    begin
      set at_end = 1;
    end;

  set LF =  chr( 10 );

  set ( lib, fil, mbr ) = ( trim( upper( lib ) ), trim( upper( fil ) ), trim( upper( mbr ) ) );
  set v_Sql = 'create or replace alias QTEMP.' concat mbr concat ' for ' concat lib concat '.' concat fil concat '(' concat mbr concat ')';
  execute immediate v_Sql;

  set v_Sql = 'select '  concat case when inc_seq = 'Y'  then 'SRCSEQ concat '','' concat ' else '' end concat
                                case when inc_date = 'Y' then 'SRCDAT concat '','' concat ' else '' end concat
              'rtrim( SRCDTA ) from QTEMP.' concat mbr;
  prepare S from v_Sql;
  open C;

  if ( trim( stmf ) <> '' ) then
    set at_end = 0;

    while ( at_end = 0 ) do
      fetch from C into v_SourceLine;
      if ( at_end = 0 ) then
        set v_StmfOut = v_StmfOut concat v_SourceLine concat LF;
        set v_LineCount = v_lineCount + 1;
      end if;
    end while;

    call QSYS2.IFS_WRITE_UTF8( PATH_NAME => stmf, LINE => coalesce( v_StmfOut, '' ), OVERWRITE => 'REPLACE', END_OF_LINE => 'NONE' );
    close C;
  end if;

  return;

end

So I went ahead and wrote an uglier one, which does everything that Christian's does, but instead reuses the already created alias:
call liama.writeMemberContent('ILEDITOR', 'ILEDITOR_TEST273_THEMEMBER', '/home/liama/ileditor_test273_member.txt');

Source
create or replace procedure liama.writeMemberContent(lib varchar(10), alias varchar(50), outfile varchar(256))
begin
  declare list varchar(20000) default '';

  declare v_Sql varchar( 128 );
  DECLARE END_C INT DEFAULT 0;

  declare LINE_DAT varchar(10) default '';
  declare LINE_TXT varchar(200) default '';

  declare c cursor for S;

  DECLARE CONTINUE HANDLER FOR NOT FOUND SET END_C = 1;

  set v_Sql = 'select SRCDAT, rtrim( SRCDTA ) as SRCDTA from ' concat lib concat '.' concat alias;

  prepare S from v_Sql;

  open c;

  FETCH c INTO LINE_DAT, LINE_TXT ;
  WHILE END_C = 0 DO
    set list = list concat LINE_DAT concat ',' concat LINE_TXT concat ux'000A';
    FETCH c INTO LINE_DAT, LINE_TXT ;
  END WHILE;

  call qsys2.ifs_write_utf8(outfile, list, overwrite => 'REPLACE');

end;

Moral of this issue, is that we need to change downloadMemberContentWithDates to call this procedure after creating the alias (we need to create the alias for reusability) and then download the resulting temporary IFS file.
I think #1968 and #1975 need to get merged before this.

P.s.
This is going to sound like a crazy idea, but we have managed to get around a bunch of CCSID issues in the database extension through the use of the Server Component. We start a single job up and we don't have to worry about encoding stuff. It's actually pretty decent. I am starting to feel more inclined to see where that fits in here as it may solve a lot of encoding issues for us in the future.

Partly solved in #1985