AWS cost attribution (feedback 1): Distinguish total AWS account cost from the 2i2c cluster attributed costs

aws-ce-grafana-backend currently reports only the costs we confidently say are associated with 2i2c managed cloud resources in the AWS account, but we know we miss out on small things.
Due to that, it would be good to show the total AWS account cost as well. For an AWS account untouched by the community, the difference should be costs not successfully attributed as 2i2c costs.
Definition of done

aws-ce-grafana-backend provides total cost data in some way
The cost attribution dashboard is able to present the total AWS account cost separate from the 2i2c attributable costs in some way that makes sense

@Gman0909 @jnywong the total costs panel is now distinguishing total account costs from total attributable costs, where the attributable costs represents almost all costs associated with 2i2c activity in a community's cloud account - but not exactly everything.
An example of a cost not captured by looking at tags as cloud infrastrucutre is use of the Cost Explorer API - it could have been used by us to present cost info in grafana or it could have been used by someone else, and we cant distinguish that by using a AWS tag like we do for everything else.

Wow that's a great job!


Invalid "optioning" of driver callbacks.

Summary
All driver callback types looks like Option<unsafe extern "system"fn (...)>, but it is wrong as it makes impossible to use them properly.
Let's look on the DRIVER_OBJECT::MajorFunction:
pub struct DRIVER_OBJECT {
    ...
    pub MajorFunction: [*mut DRIVER_DISPATCH; 28],
}

It means that MajorFunction is array of pointers to Option that holds a pointer to the handler - so, it's a pointer to pointer!
It should be something like that in such cases:
// Without `Option`:
type DRIVER_DISPATCH = extern "system" fn (...);

pub struct DRIVER_OBJECT {
    ...
    pub DriverUnload: Option<DRIVER_UNLOAD>,
    pub MajorFunction: [Option<DRIVER_DISPATCH>; 28]
}

Crate manifest
No response
Crate code
No response

I'm not familiar with this struct but it looks like a Win32 metadata issue. Will transfer for their consideration.


Standardization of layer gradient examples

#499

Except for convolution and batchnorm, all other layers are done using tf.keras.
We approach the initialization of convolution layers differently, while the functionality remains at par.

This is huge, thanks @Shashi456! 🙂


Typing in text input do not stop the event to be propagated to the game

While I type text in a zui text input, the keys are still propagated in kha resulting in things hapenning while I type.
I guess this needs some support from kha to stop propagation (a la javascript?)

You should be able to queryZui.isTyping  to stop the propagation:
https://github.com/armory3d/zui/blob/master/Sources/zui/Zui.hx#L21

Cool, that should solve my use case. Unfortunately it make the game code dependent on zui.
I wonder if we could implement something in Kha but I am not sure it has its place there

It doesn't IMO, Kha should only provide you with the very basic abstraction for everything (mouse moved here, key was pressed there). Adding something like an eventhandler (i want to bubble/cancel events) is out of scope there or we end up with something like flash.
Isn't your code tied to zui anyways as that's the nature of immediate mode?
A quick workaround would be to just add a bool somewhere in your current logic code, update that with the zui.isTyping value, and then rely on that bool instead of the zui value?

I agree, though I was not thinking of a full event handler system but just a way to allow for stopping further listener to be called. But again, this probably add unnecessary complexity for something not necessary (at least for that case)
I ll use your approach, this is actually  how I update every values via zui anyway.
Thanks


[Feature Request] Grid Flex

I love the Grid system. However I think that can be better for the UX experience create a Grid layout that can be adjust of the viewport of the browser and adjust the number of element per row.
something like devcast.co (resize the browser). Just need a little of CSS and the power of flexbox. (demo)

That layout is already using flexbox.
https://github.com/TelescopeJS/Telescope/blob/devel/packages/telescope-theme-hubble/lib/client/scss/modules/_posts.scss#L142
It would just be a matter of tweaking the existing css a little.  Would be a fairly easy pull request.  ;)

Ok, challenge accepted.  ;)
If you change these lines to the following, you get pretty close to the behavior of your examples...
.posts-grid {
	display: flex;
	flex-wrap: wrap;

	.post {
		@include large {
			width: calc((100% - 20px)/3);
			margin-right: 10px;
			&:nth-of-type(3n) {
				margin-right: 0;
			}
		}
        @include medium {
            width: calc((100% - 20px)/2);
            margin-right: 10px;
            &:nth-of-type(2n) {
                margin-right: 0;
            }
        }
        @include small {
            width: 100%;
        }

// continued...

3 wide on desktop, 2 wide on medium screens, full width on a phone.

You could just tweak min/max width and height to your preference (or create a setting for that!).

Amazing! I need some time for try this in production and I make a PR soon :-)

hmmm how can I create an another page layout based in grid that can be select from the admin panel?

I think it makes more sense to create it as a separate theme, or else change the current grid layout. It would be confusing to have two separate grid layouts.

@SachaG true :-)
@jshimko Actually I think that can be resolve better. Why limited statically the number of items? I'm thinking more in adapt with the size of the display. it's that I understand as grid flex layout.
With flex you can define the size of a element when have space and when without have, is not necessary calc the width

super simple layout avoid many unnecessary CSS and Just using flex-basis:

Looks good! I think ideally the layout would adapt to the outer container. If you don't specify anything it takes up the whole space, but if you specify width: 800px or something it shrinks down appropriately.
This way if someone wants to center their layout to get something similar to the current grid layout, they'd only need to add a single line of CSS.

I'm doing test in production, check the layout in http://zombts.com/
the code that I changed is in a separate branch in my fork (check here)
Tomorrow I want to check in more displays, but at this moment in my 13 laptop I see 5 items per row, maybe can be more beatiful more size for the elements for expect 5~7 more in large desktop.
Another question, is it necessary prefix the CSS?

Prefixing CSS is usually necessary for the end result. If we used Autoprefixer we wouldn't need to do it in the code, but I don't know if there's a Sass version for Meteor.

@SachaG ah ok, sorry, I thought that we are already using autoprefixer.

@SachaG so, why not put autoprefixer and save line of code and seconds of our lives?

We are using Autoprefixer, no?
https://github.com/TelescopeJS/Telescope/blob/devel/scss.json#L2
Via
https://atmospherejs.com/fourseven/scss

Oh you're right, completely forgot!
On Wed, Mar 11, 2015 at 4:51 AM, Jeremy Shimko<EMAIL_ADDRESS>wrote:

We are using Autoprefixer, no?
https://github.com/TelescopeJS/Telescope/blob/devel/scss.json#L2
Via
https://atmospherejs.com/fourseven/scss
—
Reply to this email directly or view it on GitHub
https://github.com/TelescopeJS/Telescope/issues/786#issuecomment-78133695
.

about this, do you want that I make a formal PR?

Yeah that'd be great! Especially if you can use the latest devel branch and see if it works with the new side nav as well :)

Hey @Kikobeats @jshimko! I'm closing this old Legacy issue, but I opened a new one totally related to it : #1464
I know you've been involved a lot in Telescope back in the days, and it's thanks to you that we can have a cool project like that nowadays!
The React version is awesome too, I hope you'll be able to give it a try 👍


socket not close when parse error

If polipo receive request like:
POST / HTTP/1.1
or
POST / HTTP/1.1
Host: www.example.com
after response with HTTP 405 error or "Couldn't parse URL ...", the connection->fd will never close (leak)

Probably the same as #34.
The issue appears to be that when notifyObject is called in httpServerRequest, there is nobody listening on the object.


Uncouple DynamicTextureAtlasBuilder from assets

Objective
Remove some unnecessary coupling between DynamicTextureAtlasBuilder and bevy_asset.
Solution
Remove the dependency of DynamicTextureAtlasBuilder::add_texture to bevy_asset, by directly passing the Image of the atlas to mutate, instead of passing separate Assets<Image> and Handle<Image> for the function to do the lookup by itself. The lookup can be done from the caller, and this allows using the builder in contexts where the Image is not stored inside Assets.
Clean-up a bit the font atlas files by introducing a PlacedGlyph type storing the GlyphId and its SubpixelOffset, which were otherwise always both passed as function parameters and the pair used as key in hash maps.
Testing
There's no change in behavior.

Changelog

Added a PlacedGlyph type aggregating a GlyphId and a SubpixelOffset. That type is now used as parameter in a few text atlas APIs, instead of passing individual values.

Migration Guide

Replace the glyph_id and subpixel_offset of a few text atlas APIs by a single place_glyph: PlacedGlyph parameter trivially combining the two.

@brandon-reinhart I'd appreciate your opinions here.

The DynamicTextureAtlasBuilder::add_texture change should be mentioned in the migration guide.

I've updated the migration guide, but one or both of you may have suggestions on how to improve this.

Migration guide looks good to me, thanks @alice-i-cecile!


chore(deps): upgrade dependencies

Upgrades project dependencies. See details in workflow run.

Automatically created by projen via the "upgrade" workflow

Codecov Report

Merging #46 (38c28e8) into main (4958138) will not change coverage.
The diff coverage is 100.00%.

@@            Coverage Diff            @@
##              main       #46   +/-   ##
=========================================
  Coverage   100.00%   100.00%
=========================================
  Files            3         3
  Lines           27        26    -1
=========================================
- Hits            27        26    -1

Impacted Files
Coverage Δ

src/backend-stack.ts
100.00% <100.00%> (ø)

src/lambda/process-metrics-function.ts
100.00% <100.00%> (ø)

src/stateful-stack.ts
100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5147bea...38c28e8. Read the comment docs.


Create Physical specification page

Should show:

spec details
flows
lineage contributions

To add later

tags  (see #1411)
aliases (?)


Windows support

msys gcc worked too

@Ciremun looks goode to me! :+1: Thank you for your contribution!


FidesCtl - Human Readable output for fides evaluate

https://ethyca.atlassian.net/jira/servicedesk/projects/FIDES/queues/custom/46/FIDES-46

@malaethyca I can't see that page, can you update the description here to include all of the relevant information?

I'll attach it here, but I think you should get access to Jira by messaging Dave. It's not practical for me to copy each customer ticket into Github.


Release notes

Fixing the empty H3 and breadcrumbs issue

Hello @ksigler7,Thanks again for the PR! It's HACKTOBERFEST! We want to show our appreciation by sending                                              you some special Hacktoberfest swag. If you have not already, could you please fill out                                              this form so we can send it to you? Thanks!                                             Team SendGrid DX


changed "infoke" to "invoke"

saw an "override operator fun infoke(navArgs: ProfileScreenNavArgs): Direction{..}" and changed it to "override operator fun invoke(navArgs: ProfileScreenNavArgs): Direction{..} at line 66

Thank you @mkaychuks 🙇♂


Feature/3493 toolbar menu api docs

Fixes # BLUI-3493

Changes proposed in this Pull Request:

Added api docs for Toolbar Menu

Screenshots:

To Test:

yarn start:reactdev

Looks good.. would be cool to get a better image for docs/src/componentDocs/ToolbarMenu/images/ToolbarMenuAnatomy.png so that the border is either not visible or it shows all 4 sides of the border.

Removed border from all the sides


add stellar test for SEDMv2

Added tests for SEDMv2 stellar pipeline.
Test:

stellar processing
stellar aperture photometry

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

3 of 3 (100.0%)  changed or added relevant lines in 3 files are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage increased (+0.05%) to 88.478%

Files with Coverage Reduction
New Missed Lines
%

mirar/processors/photometry/aperture_photometry.py
1
96.43%

Totals

Change from base Build<PHONE_NUMBER>:
0.05%

Covered Lines:
9660

Relevant Lines:
10918

💛  - Coveralls


Improved resource icons are dimmed only after a reload

Describe the bug
If I improve a resource, it should be dimmed, but that happens only when I reload the save, not immediately nor next turn.
To Reproduce
Improve a resource - it remains "not dimmed" until you reload the save
Expected behavior
Resource should became dimmed next turn
Screenshots

before improving

Several turns after resource has been improved

After reloading the save
General informations

OS: Win
Version <IP_ADDRESS>
Expansion None

ok, i think i figured this out, I'll do a pull request tomorrow

Fixed and tested.


Please offer a release as a downloadable DLL

Consumers should have other options for downloading and installing this awesome tool. Source is one option, but now thanks to asmdefs, it's easier to compile the project to a DLL and simply distribute that if users wish ( a far easier deployment model)
Or alternatively as a prepackaged asset file, already prepacked in the Plugins folder.

Hi, glad to see that you're enjoying the repo! Thanks for the info, I didn't know about asmdef, but it looks like a great feature.
Currently I'm not working with Unity / planning to work on this project, but I'm open to accepting PRs, though in this case it would be an uploaded asmdef and/or asset file. I can add a new release with these files + a small disclaimer that I wasn't able to test it.


Typo in doc after reworking PR #89

Hi @geerlingguy,
Just noticed you left one of the occurences of new variable not renamed in README.md when you reworked my PR #89.
Cheers.

Good catch, thanks for the PR!


Add recvmsg() and pktinfo support

This PR adds support for recvmsg() socket call. It also adds support for IP_PKTINFO and IPV6_RECVPKTINFO socket options. Tests for both of these features are also added.
Fixes #36415

Fixing the pktinfo tests so that we always catch if the control buffer is too small.


feat: added loader for Provider install progress

Loader to show provider install progress
Description
Added a loader for provider installation, for some feedback to the user.
Related Issue(s)
closes #734
claim #734
Screenshots
Kazam_screencast_00000.webm

Hi @rogueloop, thanks for your contribution! Code looks fine and everything works, just a few following changes should be made before we can proceed with this:
Please add appropriate Pull Request Title in the description part and upload the image directly to the github instead of the google drive link.
Also, in the Related Issue(s) part, please instead of This PR addresses issue #734 write Closes #734. /closes #734 can be removed as well.
Formatting check is failing so please tend to that as well. 😄

Addressed the issues :+1:

Added the changes pls check @idagelic.

This looks good now, thanks.
There is a check failing - I believe it is because of code formatting - please try to resolve it by running go fmt ./...

Done :heavy_check_mark:


fix: dead popup modal when opened from within fiat ramps modal

Description

Pull Request Type

[ ] :bug: Bug fix (Non-breaking Change: Fixes an issue)
[ ] :hammer_and_wrench: Chore (Non-breaking Change: Doc updates, pkg upgrades, typos, etc..)
[ ] :nail_care: New Feature (Breaking/Non-breaking Change)

Issue (if applicable)

closes #
Risk

Testing

Engineering

Operations

Screenshots (if applicable)

Current dependencies on/for this PR:

develop

PR #5238   👈

This comment was auto-generated by Graphite.


Bug: TypeError: typeorm_1.DataSource is not a constructor

Versions

Node: 16
OS: OSX

Reproduction
My project is using typeorm 0.2.34, which doesn't have DataSource so its failing inside of database/utils/synchronize

Additional Details

Steps to reproduce
run db:create with the following ormconfig.js
  database: config.POSTGRES_DATABASE,
  entities: entities,
  host: config.POSTGRES_HOST,
  migrations: migrations,
  password: config.POSTGRES_PASSWORD,
  port: config.POSTGRES_PORT,
  synchronize: false,
  migrationsRun: false,
  logging: false,
  type: 'postgres',
  username: config.POSTGRES_USER,
  uuidExtension: 'pgcrypto',
  cli: {
    migrationsDir: 'db/migrations',
  },
}

What is Expected?
Completes without error
What is actually happening?
The DB seems to be created fine, but there is a type error before the process completes. I downgraded to<EMAIL_ADDRESS>and it works without error
TypeError: typeorm_1.DataSource is not a constructor
    at /Users/joshfee/dev/x-darwin/node_modules/.pnpm/typeorm-extension@2.0.0_typeorm@0.2.34/node_modules/typeorm-extension/dist/database/utils/synchronize.js:16:28
    at Generator.next (<anonymous>)
    at /Users/joshfee/dev/x-darwin/node_modules/.pnpm/typeorm-extension@2.0.0_typeorm@0.2.34/node_modules/typeorm-extension/dist/database/utils/synchronize.js:8:71
    at new Promise (<anonymous>)
    at __awaiter (/Users/joshfee/dev/x-darwin/node_modules/.pnpm/typeorm-extension@2.0.0_typeorm@0.2.34/node_modules/typeorm-extension/dist/database/utils/synchronize.js:4:12)
    at synchronizeDatabase (/Users/joshfee/dev/x-darwin/node_modules/.pnpm/typeorm-extension@2.0.0_typeorm@0.2.34/node_modules/typeorm-extension/dist/database/utils/synchronize.js:15:12)
    at /Users/joshfee/dev/x-darwin/node_modules/.pnpm/typeorm-extension@2.0.0_typeorm@0.2.34/node_modules/typeorm-extension/dist/database/driver/postgres.js:73:51
    at Generator.next (<anonymous>)
    at fulfilled (/Users/joshfee/dev/x-darwin/node_modules/.pnpm/typeorm-extension@2.0.0_typeorm@0.2.34/node_modules/typeorm-extension/dist/database/driver/postgres.js:5:58)

@mantljosh, thx for the report.
The library version v2.0.0 should be used with typeorm version ~v0.3.0.
Do you have a recommendation where to place that piece of information?

I added a notice in the beginning of the Readme.md. I hope this is enough.
If you have a better or a suggestion in addition, please let me know ☺✌


Spurious "inherits conflicting members" errors

// Deps.scala
trait B {
  def m: Unit
}

trait C extends B {
  override def m: Unit = ???
}

trait D extends B {
  override def m: Unit = ???
}

// Test.scala
object M extends C with D

$ rsc Deps.scala

$ scalac -cp out.jar Test.scala
Test.scala:1: error: object M inherits conflicting members:
  method m in trait C of type => Unit  and
  method m in trait D of type => Unit
(Note: this can be resolved by declaring an override in object M.)
object M extends C with D
       ^

This is caused by https://github.com/scalameta/scalameta/issues/1809.


Use "current" and "future" for tox environments

This change replaces our version-specific factors for tox environments with a simpler current and future nomenclature.
This means to run the unittests on the current production version of Python, Django, and Wagtail, one would use:
tox -e unittest-current

To run them against upcoming versions of Python, Django, and Wagtail:
tox -e unittest-future

I've also added a convenience unittest environment which runs identically to unittest-current.
The intent here is to simplify the name of our test environments and make the naming version-independent.

I think you'll need to remove tox-travis, which tries to automatically select Tox environments based on the available Python (that's why it's running py36).
Since we're moving away from the tox convention in environment naming here (factors with version numbers, i.e.py36-dj111-wag23), that won't work anymore.

The ci / backend GitHub Action is failing due to black formatting.
Should I remove black from tox.ini in this Pull Request?

Should I remove black from tox.ini in this Pull Request?

Yes

Should I remove black from tox.ini in this Pull Request?

Yes

I removed black from this PR

Should I remove black from tox.ini in this Pull Request?

Yes

I removed black from this PR


Importing large databases is broken

Importing large databases seems to be broken. Only the first few tables are imported and no error is printed using CLI.
Tested using a Magento 2.3.4 database.
Might also be a bug in the error handling and displaying of errors of riptide-cli.

Issue was, that riptide-engine-docker wasn't waiting for the import to be finished. It was actually still running in the background.

Fixed via Parakoopa/riptide-engine-docker@d154519a95fdbb869ca1c087bdedf48bff2d3270


Fix bug when convert raw gyro data to rad/s

What is your source for this change?

When I use get_gyro(), the range of results looks a bit strange. Then I tried to derive it myself and found that it was a algorithm problem.

After this change, I use the result in Madgwick fusion and it works good.

How did you setup madgwick fusion and/or could you post before and after? Thanks!

How did you setup madgwick fusion and/or could you post before and after? Thanks!
Which revision of the manual did you get the image above from? Or is it some other source?

I'm sorry, I don't have a record of before and after changes. And the formula above is my own derivation. If I have misunderstood, please point it out and apologize for that.

no worries, I am not 100% sure myself.
Which madgwick filter package did you use? I'll check it out as soon as I can, prob Christmas Break

The AHRS package I use is:
[dependencies]
ahrs = "0.4"

And here is my test code: main.rs

perfekt, thank you!

Can you please still push this branch?

Can you please still push this branch to master, and push to crates.io?

I haven't been able to verify myself (yet). Can you confirm that the issue raised is correct? Not suggesting it isn't @tfx2001, I would just be more comfortable testing it out for myself, as well.

You are absolutely correct in your assessment @tfx2001.
The confusion for me arose in the way I apply "sensitivity", basically the inverse "resolution".
According to the datasheet, depending on the scale configuration of the gyroscope, the data needs to be scaled, like this:
250DPS => 250.0 / 32768.0,
500DPS => 500.0 / 32768.0,
1000DPS => 1000.0 / 32768.0,
2000DPS => 2000.0 / 32768.0,

RAW_DATA * (!) FACTOR is the default scaling. I tried to save some calculations and saved results of this scaling factor as the "sensitivity"
// 32768.0 / 250.0, ....
pub const GYRO_SENS: (f32, f32, f32, f32) = (131., 65.5, 32.8, 16.4);

of course, this "sensitvity" must be applied my dividing, exactly as you have found.
Thanks again, I will merge and publish to crates.io.

done


Fix a bug in memory laws with redundant also

This adds the tests in #645, verifies that bug manifests, then I fixed the bug. Seems to fix #631 for real. /cc @pankajroark

It also fixes #646

LGTM, maybe minor comment update if the revision makes sense.


1.12.2 version?

I thought I finally found a fix for this issue but nobody cares about 1.12.2 anymore :(
I understand why, 1.12.2 is considered abandoned at this point. Many players such as me cannot update to newer versions.
Could you please make just one backport for 1.12.2? I doubt Xaero themselves would fix the issue, since 1.12.2 is abandoned.

xaero's minimap for 1.12.2 is still being updated. please make an issue on xaero's issue tracker as i'd like to stop maintaining this mod, let alone port it to different versions.


Add @type support to paths in Go

Fab idea. PTAL.

Diff Coverage
File Path

•••••••••• 100%
go/types/path.go

Powered by Codecov. Last update fcb98d7...ace3359


Paypal balance broken

In the host view on the expenses page, it says "available balance", a number, and a button under to refill the Paypal preapproval at the top right of the page. However, this number is not correct and does not update as expected.

This is still an issue.

@xdamman were you working on this? It's half fixed - now when you load the dashboard it shows the correct preapproval balance, it just doesn't update when you make a payment and you have to refresh to update.

@Betree has been working on this

@alannallama Indeed I fixed the balance on the API side but we don't refresh this data on the frontend when we pay expenses. I've moved this part to a new issue: https://github.com/opencollective/opencollective/issues/1855


PR for auth/README.md code example fix.

I have a small patch for  auth/README.md, looks like getIntent().getExtras().getString(ExtraConstants.EMAIL_LINK_SIGN_IN) doesn't work anymore, I had to change it to getIntent().getData().toString() and also there's an error in the null check of the example.
Here's how firebase auth gets the link internally:
https://github.com/firebase/FirebaseUI-Android/blob/6d41fb2992a7ce01d7f38e8451805e49e96e5628/auth/src/main/java/com/firebase/ui/auth/AuthUI.java#L238
My patch is here:
https://github.com/carrizo/FirebaseUI-Android/commit/db9507c1e0a97352755a72f71d50c67b2c954584
Does it look fine for a PR?

@carrizo thanks for asking!  Happy to accept your PR.


Fix for chmod 777

Task name:  ContainerBuildV0, ContainerStructureTestV0, DockerInstallerV0, HelmInstallerV0, KubernetesV0,
KubernetesV1, PackerBuildV0, PackerBuildV1
Description:  Dear task owners! An automatic security check raised an alert about "The Microsoft code repository [repository link] sets world writeable permissions to files/folders, which later are executed. This results in Elevation of Privilege (EoP) that occurs when a local user with limited permissions overwrites the world writeable files/folders between the time permissions are set and a file is executed."
It seems to me that exploitability of this is not very big, however I believe we have to check this with task owners.
Checklist:

[ ] Task version was bumped - please check instruction how to do it
[ ] Checked that applied changes work as expected

/azp run

@mpodriezov Out of these tasks ContainerBuildV0, ContainerStructureTestV0, DockerInstallerV0, HelmInstallerV0, KubernetesV0, KubernetesV1, PackerBuildV0, PackerBuildV1 we own only 5 tasks [ContainerBuildV0, DockerInstallerV0, HelmInstallerV0, KubernetesV0, KubernetesV1].
We have validated these tasks and they are working fine with 644 permissions.
We are approving for these 5 tasks please check with corresponding team for other tasks validations.
Thanks

For ContainerStructureTestV0 task:

644 will cause below error, we require executable permissions also. Hence 744 will work instead of 644.
error details:

##[error]Error: Unable to locate executable file: '/opt/hostedtoolcache/container-structure-test/1.0.0/x64/container-structure-test'. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also check the file mode to verify the file is executable.
https://dev.azure.com/triptijain/1838367/_build/results?buildId=242&view=logs&j=275f1d19-1bd8-5591-b06b-07d489ea915a&t=b4b48ea1-9780-50dc-a419-857113b35200

Please bump task version also to know there were some change/update in task.


How to set time zone offset?

ahhh, I know this is a naive question.  but different date library has different rule, I usually use yyyy-MM-dd'T'HH:mm:ss+08  in other date libraries, but it doesn't work for this library .  I'm a little confused.
i use a logging library named pino, It uses this library, and time zone is UTC+0, and I want to change it.
So i try to find answer in your document, and I found the following Setting, it seems to be what i want, but I don't know how to use it.

~
~

o
GMT/UTC timezone offset, e.g. -0500 or +0230.

What should I do?

I have this same issue. Looking through the issues list now to see if it's been solved

@light0x00 I also use pino with pino-pretty.  The o mask pattern just prints out the current offset.  It doesn't set an offset and change the output to local time if it was UTC.  I wanted the log to display the time with local time zone.  My app runs in docker so I achieved this with pino-pretty by doing the following:

set the environment variable -e "TZ=US/Central" on my docker container
set the pino-pretty option "translateTime" using the SYS: prefix "SYS:yyyy-mm-dd hh:MM:ss TT Z"

The timestamp entry ended up looking like this afterwards:
[2020-03-01 07:49:02 PM CST]


Numbering of Delay condonation application

App Numbering _ Delay Condonation.pdf

We have scope changes to the feature so we are dropping this feature from this sprint

Hi All moving this ticket to next sprint due to scope is not finalized

@manimaarans @Ramu-kandimalla @Susmitabe @suresh12 @Taherabharmal  please note
delay condonation workflow - https://github.com/pucardotorg/dristi-solutions/tree/develop/docs/asyncsubmission/workflow
delay condonation sequence diagrams - https://github.com/pucardotorg/dristi-solutions/tree/develop/docs/UI_Sequence_Diagrams/Case Flows

Tested in dev and it will be pushed to qa along with all other changes in develop


[BUG] "delete" does not work

Is there an existing issue for this?

[X] I have searched the existing issues

Does this issue exist in the latest version?

[X] I'm using the latest release

Describe the bug?
Deleted files in a sub directory are not listed in Deleted files in logs and any_modified returns false
To Reproduce

create new repo
add workflow file to .github/workflows
create testdir on root directory
add testfile1 and testfile2 to testdir
commit
delete testfile2
commit

What OS are you seeing the problem on?
ubuntu-latest or ubuntu-20.04
Expected behavior?
When I delete file in testdir/, the result is true
jobs:

  check:
    runs-on: ubuntu-latest
    steps:

      - uses: actions/checkout@v2
        with:
          fetch-depth: 0

      - name: Is Modified?
        id: modified
        uses<EMAIL_ADDRESS>        with:
          files: |
            testdir/**

      - name: Done!
        run: |
          echo Modified ${{ steps.modified.outputs.any_modified }}

Relevant log output
2022-02-10T09:21:36.4636390Z Requested labels: ubuntu-latest
2022-02-10T09:21:36.4636439Z Job defined at: n2i-inc/test-tj-acttions-changed-files/.github/workflows/main.yml@refs/heads/main
2022-02-10T09:21:36.4636464Z Waiting for a runner to pick up this job...
2022-02-10T09:21:36.7825507Z Job is waiting for a hosted runner to come online.
2022-02-10T09:21:41.3593023Z Job is about to start running on the hosted runner: Hosted Agent (hosted)
2022-02-10T09:21:44.6625127Z Current runner version: '2.287.1'
2022-02-10T09:21:44.6655546Z ##[group]Operating System
2022-02-10T09:21:44.6656447Z Ubuntu
2022-02-10T09:21:44.6656735Z 20.04.3
2022-02-10T09:21:44.6657186Z LTS
2022-02-10T09:21:44.6657484Z ##[endgroup]
2022-02-10T09:21:44.6657883Z ##[group]Virtual Environment
2022-02-10T09:21:44.6658328Z Environment: ubuntu-20.04
2022-02-10T09:21:44.6658683Z Version: 20220207.1
2022-02-10T09:21:44.6659969Z Included Software: https://github.com/actions/virtual-environments/blob/ubuntu20/20220207.1/images/linux/Ubuntu2004-Readme.md
2022-02-10T09:21:44.6660613Z Image Release: https://github.com/actions/virtual-environments/releases/tag/ubuntu20%2F20220207.1
2022-02-10T09:21:44.6661225Z ##[endgroup]
2022-02-10T09:21:44.6661520Z ##[group]Virtual Environment Provisioner
2022-02-10T09:21:44.6661897Z <IP_ADDRESS>-main-20220202-1
2022-02-10T09:21:44.6662253Z ##[endgroup]
2022-02-10T09:21:44.6663141Z ##[group]GITHUB_TOKEN Permissions
2022-02-10T09:21:44.6663763Z Actions: write
2022-02-10T09:21:44.6664330Z Checks: write
2022-02-10T09:21:44.6664678Z Contents: write
2022-02-10T09:21:44.6665003Z Deployments: write
2022-02-10T09:21:44.6665275Z Discussions: write
2022-02-10T09:21:44.6665603Z Issues: write
2022-02-10T09:21:44.6665896Z Metadata: read
2022-02-10T09:21:44.6666210Z Packages: write
2022-02-10T09:21:44.6666466Z Pages: write
2022-02-10T09:21:44.6666786Z PullRequests: write
2022-02-10T09:21:44.6667084Z RepositoryProjects: write
2022-02-10T09:21:44.6667470Z SecurityEvents: write
2022-02-10T09:21:44.6667818Z Statuses: write
2022-02-10T09:21:44.6668079Z ##[endgroup]
2022-02-10T09:21:44.6672603Z Secret source: Actions
2022-02-10T09:21:44.6673249Z Prepare workflow directory
2022-02-10T09:21:44.7688281Z Prepare all required actions
2022-02-10T09:21:44.7966378Z Getting action download info
2022-02-10T09:21:44.9672270Z Download action repository 'actions/checkout@v2' (SHA:ec3a7ce113134d7a93b817d10a8272cb61118579)
2022-02-10T09:21:45.3782373Z Download action repository<EMAIL_ADDRESS>(SHA:2c5f9483f5e7738915d7c70d18ad25c41c6e4ff4)
2022-02-10T09:21:45.7443906Z Getting action download info
2022-02-10T09:21:45.8830486Z Download action repository 'tj-actions/glob@v6' (SHA:7442992c17785425e88a1c0ae2db029cd8458740)
2022-02-10T09:21:46.1034941Z Getting action download info
2022-02-10T09:21:46.2291624Z Download action repository 'actions/github-script@v5' (SHA:e3cbab99d3a9b271e1b79fc96d103a4a5534998c)
2022-02-10T09:21:46.5653435Z ##[group]Run actions/checkout@v2
2022-02-10T09:21:46.5653940Z with:
2022-02-10T09:21:46.5654129Z   fetch-depth: 0
2022-02-10T09:21:46.5654400Z   repository: n2i-inc/test-tj-acttions-changed-files
2022-02-10T09:21:46.5655027Z   token: ***
2022-02-10T09:21:46.5655228Z   ssh-strict: true
2022-02-10T09:21:46.5655459Z   persist-credentials: true
2022-02-10T09:21:46.5655657Z   clean: true
2022-02-10T09:21:46.5655862Z   lfs: false
2022-02-10T09:21:46.5656054Z   submodules: false
2022-02-10T09:21:46.5656272Z ##[endgroup]
2022-02-10T09:21:46.8941219Z Syncing repository: n2i-inc/test-tj-acttions-changed-files
2022-02-10T09:21:46.9075801Z ##[group]Getting Git version info
2022-02-10T09:21:46.9078571Z Working directory is '/home/runner/work/test-tj-acttions-changed-files/test-tj-acttions-changed-files'
2022-02-10T09:21:46.9080505Z [command]/usr/bin/git version
2022-02-10T09:21:46.9112732Z git version 2.35.1
2022-02-10T09:21:46.9129311Z ##[endgroup]
2022-02-10T09:21:46.9139655Z Deleting the contents of '/home/runner/work/test-tj-acttions-changed-files/test-tj-acttions-changed-files'
2022-02-10T09:21:46.9144971Z ##[group]Initializing the repository
2022-02-10T09:21:46.9149821Z [command]/usr/bin/git init /home/runner/work/test-tj-acttions-changed-files/test-tj-acttions-changed-files
2022-02-10T09:21:46.9253097Z hint: Using 'master' as the name for the initial branch. This default branch name
2022-02-10T09:21:46.9253871Z hint: is subject to change. To configure the initial branch name to use in all
2022-02-10T09:21:46.9257610Z hint: of your new repositories, which will suppress this warning, call:
2022-02-10T09:21:46.9258642Z hint:
2022-02-10T09:21:46.9259712Z hint: 	git config --global init.defaultBranch <name>
2022-02-10T09:21:46.9260808Z hint:
2022-02-10T09:21:46.9262066Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2022-02-10T09:21:46.9262810Z hint: 'development'.
2022-02-10T09:21:47.3920721Z # "Set base sha..."
2022-02-10T09:21:47.3920933Z if [[ -n "" ]]; then
2022-02-10T09:21:47.3921192Z   echo "::set-output name=base_sha::"
2022-02-10T09:21:47.3921437Z elif [[ "false" == "true" ]]; then
2022-02-10T09:21:47.3921693Z   if [[ "" != "0000000000000000000000000000000000000000" ]]; then
2022-02-10T09:21:47.3922149Z     echo "::set-output name=base_sha::"
2022-02-10T09:21:47.3922444Z   else
2022-02-10T09:21:47.3922723Z     echo "::set-output name=base_sha::a52953944a1cecfaada86d9b7ac07450f12f8cf3"
2022-02-10T09:21:47.3923020Z   fi
2022-02-10T09:21:47.3923340Z fi
2022-02-10T09:21:47.3994328Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-02-10T09:21:47.3995471Z ##[endgroup]
2022-02-10T09:21:47.4254760Z ##[group]Run tj-actions/glob@v6
2022-02-10T09:21:47.4255023Z with:
2022-02-10T09:21:47.4255207Z   files: testdir/**

2022-02-10T09:21:47.4255446Z   files-separator:

2022-02-10T09:21:47.4255644Z   separator: |
2022-02-10T09:21:47.4256022Z   token: ***
2022-02-10T09:21:47.4256442Z   files-from-source-file-separator:

2022-02-10T09:21:47.4256698Z   follow-symbolic-links: true
2022-02-10T09:21:47.4256946Z   strip-top-level-dir: true
2022-02-10T09:21:47.4257147Z ##[endgroup]
2022-02-10T09:21:47.4305440Z ##[group]Run actions/github-script@v5
2022-02-10T09:21:47.4305668Z with:
2022-02-10T09:21:47.4306060Z   github-token: ***
2022-02-10T09:21:47.4307191Z   script: const path = require("path");

const { GITHUB_ACTION_PATH } = process.env
const ACTION_PATH = path.join(GITHUB_ACTION_PATH, "main.js")
const FILES_FROM_SOURCE_FILE_PATH = path.join(GITHUB_ACTION_PATH, "getFilesFromSourceFile.js");
const UTILS_PATH = path.join(GITHUB_ACTION_PATH, "util.js");

const { getFilesFromSourceFile } = require(FILES_FROM_SOURCE_FILE_PATH);
const { normalizeSeparators } = require(UTILS_PATH);
const main = require(ACTION_PATH)

await main({ core, glob, getFilesFromSourceFile, normalizeSeparators })

2022-02-10T09:21:47.4308111Z   debug: false
2022-02-10T09:21:47.4308316Z   user-agent: actions/github-script
2022-02-10T09:21:47.4308735Z   result-encoding: json
2022-02-10T09:21:47.4308953Z env:
2022-02-10T09:21:47.4309195Z   GITHUB_ACTION_PATH: /home/runner/work/_actions/tj-actions/glob/v6
2022-02-10T09:21:47.4309607Z   GITHUB_WORKSPACE: /home/runner/work/test-tj-acttions-changed-files/test-tj-acttions-changed-files
2022-02-10T09:21:47.4309942Z   INPUT_FILES: testdir/**

2022-02-10T09:21:47.4310180Z   INPUT_FILES_FROM_SOURCE_FILE:
2022-02-10T09:21:47.4310416Z   INPUT_FILES_SEPARATOR:

2022-02-10T09:21:47.4310649Z   INPUT_FILES_FROM_SOURCE_FILE_SEPARATOR:

2022-02-10T09:21:47.4310910Z   INPUT_FOLLOW_SYMBOLIC_LINKS: true
2022-02-10T09:21:47.4311124Z   INPUT_SEPARATOR: |
2022-02-10T09:21:47.4311354Z   INPUT_STRIP_TOP_LEVEL_DIR: true
2022-02-10T09:21:47.4311553Z ##[endgroup]
2022-02-10T09:21:47.5432649Z
2022-02-10T09:21:47.5458250Z
2022-02-10T09:21:47.5646721Z ##[group]Run bash $GITHUB_ACTION_PATH/entrypoint.sh
2022-02-10T09:21:47.5647243Z bash $GITHUB_ACTION_PATH/entrypoint.sh
2022-02-10T09:21:47.5744519Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2022-02-10T09:21:47.5744810Z env:
2022-02-10T09:21:47.5745026Z   GITHUB_SERVER_URL: https://github.com
2022-02-10T09:21:47.5745340Z   GITHUB_REPOSITORY: n2i-inc/test-tj-acttions-changed-files
2022-02-10T09:21:47.5745605Z   GITHUB_BASE_REF:
2022-02-10T09:21:47.5746040Z   INPUT_SHA: a52953944a1cecfaada86d9b7ac07450f12f8cf3
2022-02-10T09:21:47.5746305Z   INPUT_BASE_SHA:
2022-02-10T09:21:47.5746797Z   INPUT_TOKEN: ***
2022-02-10T09:21:47.5747056Z   INPUT_FILES_PATTERN: testdir|testdir/testfile1
2022-02-10T09:21:47.5747301Z   INPUT_SEPARATOR:
2022-02-10T09:21:47.5747484Z   INPUT_PATH:
2022-02-10T09:21:47.5747682Z ##[endgroup]
2022-02-10T09:21:47.5869476Z ##[group]changed-files
2022-02-10T09:21:47.5869985Z Resolving repository path...
2022-02-10T09:21:47.5908254Z Setting up 'temp_changed_files' remote...
2022-02-10T09:21:47.5957289Z No 'temp_changed_files' remote found
2022-02-10T09:21:47.5957990Z Creating 'temp_changed_files' remote...
2022-02-10T09:21:47.6000323Z Getting HEAD info...
2022-02-10T09:21:47.6071502Z Retrieving changes between 73b159cb1d5a7fc80481c10f771c17ae2f6a3f50 (main) → a52953944a1cecfaada86d9b7ac07450f12f8cf3 (main)
2022-02-10T09:21:47.6072403Z Getting diff...
2022-02-10T09:21:47.6073270Z Input files pattern: testdir|testdir/testfile1
2022-02-10T09:21:47.6793508Z Non Matching modified files: testdir/testfile2
2022-02-10T09:21:47.6902558Z Non Matching deleted files: testdir/testfile2
2022-02-10T09:21:47.7330999Z Added files:
2022-02-10T09:21:47.7331469Z Copied files:
2022-02-10T09:21:47.7332070Z Deleted files:
2022-02-10T09:21:47.7332633Z Modified files:
2022-02-10T09:21:47.7333212Z Renamed files:
2022-02-10T09:21:47.7333788Z Type Changed files:
2022-02-10T09:21:47.7334568Z Unmerged files:
2022-02-10T09:21:47.7335099Z Unknown files:
2022-02-10T09:21:47.7335662Z All changed and modified files:
2022-02-10T09:21:47.7336047Z All changed files:
2022-02-10T09:21:47.7336811Z All modified files:
2022-02-10T09:21:47.7364119Z ##[endgroup]
2022-02-10T09:21:47.7401074Z ##[group]Run echo Modified false
2022-02-10T09:21:47.7401556Z echo Modified false
2022-02-10T09:21:47.7460997Z shell: /usr/bin/bash -e {0}
2022-02-10T09:21:47.7461226Z ##[endgroup]
2022-02-10T09:21:47.7536708Z Modified false
2022-02-10T09:21:47.7590935Z Post job cleanup.
2022-02-10T09:21:47.9000284Z [command]/usr/bin/git version
2022-02-10T09:21:47.9065358Z git version 2.35.1
2022-02-10T09:21:47.9126701Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2022-02-10T09:21:47.9174562Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2022-02-10T09:21:47.9439446Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2022-02-10T09:21:47.9466015Z http.https://github.com/.extraheader
2022-02-10T09:21:47.9476102Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2022-02-10T09:21:47.9510993Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2022-02-10T09:21:47.9962376Z Cleaning up orphan processes

Anything else?
I had test this behavior in my repo.
Please check this result, test data, and workflow.
https://github.com/n2i-inc/test-tj-acttions-changed-files/
https://github.com/n2i-inc/test-tj-acttions-changed-files/actions/runs/1822907572
Thank you.
(I'm sorry for my poor English)
Code of Conduct

[X] I agree to follow this project's Code of Conduct

After investigating this issues seems to occur when glob patterns are used and a short terms solution is to omit the files input
Example:
      - name: Is Modified?
        id: modified
        uses<EMAIL_ADDRESS>

@jackton1
Thank you for your reply.
In my repository, the API and frontend have separate directories (/api and /frontend), and I want to separate the behavior for each. So it's difficult to omit the directory.
I would be very happy if you could correct it.

@n2i-onimaru This should be resolved now in the latest version. Thanks


_laser_lock_box doesn't accept negative gain values

I would like to invert the sign of the slow controls PID in the laser lock box.  The set_pid_by_gain helper method will not accept negative gains for the 'g' parameter or for the I or D components.  This is available in iPad interface, so it is possible, but missing the python API.
I've had some email correspondence with LI team.  But its probably easier to have proper issue tracking here.
For the code version distributed by pip (as of Dec @2019) there is an assert that throws and error.  The set_pid_by_gain function has an assert to prevent negative g values, line 327:
_utils.check_parameter_valid('range', g, [0, 2**16 - 1], desc='Gain', units='linear scalar')

After some email exchange with LI team I changed this to:
_utils.check_parameter_valid('range', g, [-2**16, 2**16 - 1], desc='Gain', units='linear scalar')

This fixes the initial problem but there are some sign issues deeper into the code that is probably beyond me to fix in a quick hack.  Below is some sample code (and my terminal output) that is hopefully reproducible:
# Moku access
from pymoku import Moku
from pymoku.instruments import LaserLockBox
from scipy import signal

def gen_butterworth(corner_frequency):
    """
    Generate coefficients for a second order butterworth low-pass filter.

    Corner frequencies for laser lock box second harmonic filtering should be
    in the range: 1 kHz < corner frequency < 31.25 MHz.
    """
    sample_rate = 31.25e6
    normalised_corner = corner_frequency / (sample_rate / 2)
    b, a = signal.butter(2, normalised_corner, 'low', analog=False)

    coefficient_array = [[1.0, b[0], b[1], b[2], -a[1], -a[2]],
                         [1.0, 1.0, 0.0, 0.0, 0.0, 0.0]]
    return coefficient_array

m = Moku.get_by_name('Space-Black')
i = m.deploy_or_connect(LaserLockBox)

#set input config
i.set_frontend(1, ac=True)
i.set_input_gain(gain=0)

# set enables
i.set_output_enables(1, True)
i.set_output_enables(2, True)
i.set_pid_enables(1, False)
i.set_pid_enables(2, False)
i.set_channel_pid_enables(1, False)
i.set_channel_pid_enables(2, False)

# set local oscillator, auxiliary and scan generators
i.set_local_oscillator(source='internal', frequency=3.0e6, phase=231.0, pll_auto_acq=False)
i.set_aux_sine(amplitude=0.500, frequency=3.0e6, phase=0.0, sync_to_lo=True, output='out2')
i.set_scan(frequency=10.e0, phase=0, output='out1', amplitude=1.0, waveform='sawtooth')

# configure second harmonic rejection low pass filter
coef_array = gen_butterworth(200.0e3)
i.set_custom_filter(coef_array)

# Attempt at setting initial locking state, si=100.0 to set Int saturation at two decades below zero crossing of intergrator
i.set_pid_by_freq(1, kp=0.0, i_xover=1500.0, d_xover=None, si=10.0, enable=True)
i.set_pid_by_gain(2, g=-1.0, kp=0.0, ki=1500.0, kd=0, si=None, sd=None, enable=False)

# set offsets
i.set_offsets(position='pid_input', offset=0.0)
i.set_offsets(position='out1', offset=0.00)
i.set_offsets(position='out2', offset=0.00)

# set allowable output range
i.set_output_range(1, 1.0, -1.0)
i.set_output_range(2, 0.5, -0.5)

# Monitor the error signal and fast pid output signal
i.set_monitor('A', 'error')
i.set_monitor('B', 'out1')  # green
i.set_precision_mode(True)

# Trigger on rising edge of the scan signal, 0V threshold level with 0.1V
# hysteresis
i.set_trigger('scan', 'rising', level=0, hysteresis=100.0e-6,
              trig_on_scan_rising=True)

# View +- 2 millisecond, i.e. trigger in the centre
i.set_timebase(-0.1, 0.1)

# Get initial data frame to set up plotting parameters. This can be done
# once if we know that the axes aren't going to change (otherwise we'd do
# this in the loop)
data = i.get_realtime_data()

and the return I get back from python...
ValueOutOfRangeException                  Traceback (most recent call last)
<ipython-input-4-17e1b802495d> in <module>
     22 # Attempt at setting initial locking state, si=100.0 to set Int saturation at two decades below zero crossing of intergrator
     23 i.set_pid_by_freq(1, kp=0.0, i_xover=1500.0, d_xover=None, si=10.0, enable=True)
---> 24 i.set_pid_by_gain(2, g=-1.0, kp=0.0, ki=1500.0, kd=0, si=None, sd=None, enable=False)
     25
     26 # set offsets

~\Anaconda3\lib\site-packages\pymoku\_instrument.py in wrapper(self, *args, **kwargs)
    263                         try:
    264                                 # Attempt to call the wrapped function
--> 265                                 res = func(self, *args, **kwargs)
    266                         finally:
    267                                 # Do this even if the function raises an Exception

~\Anaconda3\lib\site-packages\pymoku\_laser_lock_box.py in set_pid_by_gain(self, pid_block, g, kp, ki, kd, si, sd, enable)
    347                         self.fast_pid_en = enable
    348                 else:
--> 349                         self.slow_pid.set_reg_by_gain(g, kp, ki, kd, si, sd)
    350                         self.slow_pid.gain = self.slow_pid.gain * 2**15
    351                         self.slow_pid_en = enable

~\Anaconda3\lib\site-packages\pymoku\_pid.py in set_reg_by_gain(self, g, kp, ki, kd, si, sd)
    182                 # ADC, DAC, decimation gains, etc. are not accounted for here))
    183
--> 184                 self.gain = g
    185                 self.p_gain = kp
    186                 self.i_gain = ki / self.ang_freq

~\Anaconda3\lib\site-packages\pymoku\_pid.py in gain(self, value)
    105         def gain(self, value):
    106                 r = self.reg_base + PID._REG_GAIN
--> 107                 self._instr._accessor_set(r, to_reg_unsigned(0, 32), value)
    108
    109         @property

~\Anaconda3\lib\site-packages\pymoku\_instrument.py in _accessor_set(self, reg, set_xform, data)
    319                         old = self._localregs[reg] if self._localregs[reg] is not None else self._remoteregs[reg] or 0
    320
--> 321                 new = set_xform(self, data, old)
    322                 if new is None:
    323                         raise ValueOutOfRangeException("Reg %d Data %d" % (reg, data))

~\Anaconda3\lib\site-packages\pymoku\_instrument.py in __us(obj, val, old)
    156                         return None
    157
--> 158                 v = _usgn(val, _len) << _offset
    159
    160                 try:

~\Anaconda3\lib\site-packages\pymoku\_instrument.py in _usgn(i, width)
     64                 return int(i)
     65
---> 66         raise ValueOutOfRangeException("%d doesn't fit in %d unsigned bits" % (i, width))
     67
     68 def _sgn(i, width):

ValueOutOfRangeException: -1 doesn't fit in 32 unsigned bits

Unresolved but no response so closing.


Delete / document reappears when searching issue

Hello,
Our issue is as follows:
We are trying to delete a single document. It should be ok according to the response of the request:
Request:
curl -XDELETE 'https://our_url_here/our_index_here/_doc/d83bf9d0-99b0-11e9-8d94-fdd299fdd95f'

Response:
{"_index":"our_index_here","_type":"_doc","_id":"d83bf9d0-99b0-11e9-8d94-fdd299fdd95f","_version":1,"result":"deleted","_shards":{"total":1,"successful":1,"failed":0}}

It would appear to also work when searching (the document is no longer included in search results), but after a few seconds it's being included again:
Immediate response (document no longer shows up):
curl -XGET 'https://our_url_here/our_index_here/_search?pretty=true'
{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "our_index_here",
        "_type" : "_doc",
        "_id" : "3c3dc610-99a5-11e9-84f8-f3f4655d3ad9",
        "_score" : 1.0
      }
    ]
  }
}

After searching a few seconds later (you can see that the document with the deleted id starts showing up again):
curl -XGET 'https://our_url_here/our_index_here/_search?pretty=true'
{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 2,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "our_index_here",
        "_type" : "_doc",
        "_id" : "3c3dc610-99a5-11e9-84f8-f3f4655d3ad9",
        "_score" : 1.0,
      },
      {
        "_index" : "our_index_here",
        "_type" : "_doc",
        "_id" : "d83bf9d0-99b0-11e9-8d94-fdd299fdd95f",
        "_score" : 1.0
      }
    ]
  }
}

What could be causing this issue ? I know that documents are not deleted on the spot (only marked for delete) but should they not be excluded from search results ?
We are using version <IP_ADDRESS>.
Looking forward to your answer.
Thanks.

Hi,
If you have multiple nodes, with a replication factor > 1, if a tombestone is not properly replicated, you might get this result from a node where the data is not yet deleted. You should enable TRACE for org.elassandra.index, and check the logs/system.logs on your nodes.
Thanks.

Hello,
Thanks for answering.
We only have one node though. Could this be a sync issue between cassandra and elasticsearch ?

After enabling TRACE level logging, we found some info that will help us solve this issue.
Thank you for your time.


[python] Initial work toward PyTorch data loaders

This PR is superceded by TileDB-Inc/TileDB-SOMA-ML#1 and is being closed.  No further work will be done on this branch.

Notes to reviewers:

APIs have changed from the CZI contribution. The demo notebook (tutorial_pytorch.ipynb) has been updated to run with all API modifications.
This PR is for an initial drop we plan to use for pre-release feedback. More work is required prior to release
This is set up as a separate Python package, with its own config/build/etc.

This PR contains a PyTorch iterable-style DataSet/DataPipe for use with SOMA Experiment. Initial code contributed by the Chan Zuckerberg single-cell team. Modifications to the original code include:

stand-alone Python package tiledbsoma_ml
significant refactoring to improve performance
CI and other config for stand-alone Python package
improved context/config handling  - pass through all user-specified configuration allowing use with other storage solutions, etc.
fix lint and typing issues identified by the CI pipeline in this repo
update doctrings and copyright
some minor enhancements to unit tests
added torch.utils.data.IterableDataset alongside the existing torchdata.datapipes.iter.IterDataPipe, acknowledging that the torchdata.datapipes is deprecated and slated for removal
add Python 3.8 support
API refinement to better align with ExperimentAxisQuery
modifications to support DDP (mult-GPU) training.

API change summary:

constructor takes ExperimentAxisQuery, rather than than an Experiment
reworked I/O and shuffle chunk size param for better API UX

Benchmarking results vs the cellxgene-census loader will be forthcoming shortly.

PR superceded by TileDB-Inc/TileDB-SOMA-ML#1


Add support for the functions listxattr and removexattr

In order to create FUSE filesystems which properly support extended attributes, the functions listxattr and removexattr are required in addition to setxattr and getxattr. This patch brings in those functions, which already exist in libfuse.

2.11.0


Vagrant not exiting with non-zero status

2015-11-05 00:21:46,477 14 [DEBUG] -  [Vagrant] ==> default: Exiting with 1
2015-11-05 00:21:46,533 5 [DEBUG] - Command ['"C:\HashiCorp\Vagrant\bin\vagrant.exe" provision'] exited with '0'

https://github.com/mitchellh/vagrant/issues/6549


[Bug]: 从windows微信客户端@机器人，msg.mention_text() 会报错，手机端不会

Environment
- wechaty:
- wechaty-puppet:
- wechaty-puppet-service:
- wechaty-plugin-contrib:
- token type: padlocal|wxwork|xp
- the version of wechaty docker container: [0.65]

Description
从windows微信客户端@机器人，msg.mention_text() 会报错

Wechaty - ERROR - internal error <>

Minimum reproducible code
text = await msg.mention_text()

这里contact_id会拿到一个空""，可以先判空过滤一下绕过
https://github.com/wechaty/python-wechaty/blob/6c9d54ffc2ad4042fdb000c417b90a6807dfc29b/src/wechaty/user/message.py#L462

It's not a bug in python-wechaty: you should check whether the text is mentional message with the following:
if await msg.mention_self():
    pass

if await msg.mention_list():
    mention_text = await msg.mention_text()

So please help yourself to upgrade the code. Good luck to you.

同样遇到这个问题，通过
        if await msg.mention_self():
            if await msg.mention_self():
                pass
            if await msg.mention_list():
                keyword = await msg.mention_text()

这个代码没有解决。
应用https://github.com/wechaty/python-wechaty/pull/390这个pr，可以解决问题，希望官方可以重视。

According to the diffs between https://github.com/wechaty/python-wechaty/pull/390 and https://github.com/wechaty/python-wechaty/tree/main, it seems that cached contacts is dirty not been updated.  Fetching all of contacts works but violence.
I will try to merge this pr to fix this bug and refactor it later.


-e ignore and -e kill for more error-handling options

https://www.gnu.org/software/make/manual/html_node/Errors.html
We now have -k (#1265 ) for --keep-going. GNU Make has an similar option that we can potentially borrow.We could also allow option allow_errors in more places .

Not sure how that's going to differ from -k in essence; other than possibly it is not going to do the atomic write that removes output, if exists & return non-zero status? If so, the behavior is not good ...

I found a possible application of option. In my data process pipeline, sometimes different parameters are given to skip parts of the pipeline. This results in datasets with "holes" here and there, causing difficulties in the "summarize" step. Although something like
[default]
if fileA.exists():
   sos_run('process_fileA')
if fileB.exists():
   sos_run('process_fileB')

[process_fileA]
[process_fileB]

will work, the need to figure out what file_A should be can be a burden, and it might be easier just to run the pipeline that runs through when process_fileA and/or process_fileB fails.

Okay I'm digging up this old open ticket to continue discussion here for error handling behavior. This is currently an issue with running large benchmark workflows when only a few substeps go wrong. We need two behavior:

an --ignore-errors behavior that creates dummy outputs eg with zapped extension, for example, for substeps that contains the error. That case the workflow will always run successfully but when we see zapped files there we know there was an error (and of course errors will be documented into stderr files if they are specified). Here I discuss with a toy example:
some behavior to stop a step immediately if any substep fails. I think it was previously the default behavior but we changed it to currently wait for completion of the entire step.

@BoPeng do you think these are reasonable requests that we currently dont have available? Command interface would be a next thing to worry about but I would like to make sure I'm correct about current status.
Here is a toy example we can use to test out the behavior:
[global]
parameter: sec = 2
parameter: n = 5
reps = [i for i in range(n)]

[a]
input: for_each = 'reps'
output: f'a{_reps}.txt'
if _index == 3:
	raise ValueError
else:
	run(f'sleep {sec} && echo a{_index} > {_output}')
[b]
input: for_each = 'reps'
output: f'b{_reps}.txt'
run(f'sleep {sec * 3} && echo b{_index} > {_output}')

[default_1]
depends: sos_step('a')
input: None
output: [f'a{i}.txt' for i in range(n)]

[default_2]
depends: sos_step('b')
input: None
output: [f'b{i}.txt' for i in range(n)]

[clean]
files = [f'a{i}.txt' for i in range(n)] + [f'b{i}.txt' for i in range(n)]
bash: expand = True
	rm -f {' '.join(files)}

Current behaviors
Default behavior:
sos run keep-going.sos clean && sos run keep-going.sos -j10

output:
...
WARNING: Substep a (id=c32cfc1964fed41a, index=3) returns an error. Terminating step after completing 3 submitted substeps.
...
ValueError:
[default]: 2 pending steps: default_1, default_2

Keep going behavior:
sos run keep-going.sos clean && sos run keep-going.sos -j 10 -k

output:
...
WARNING: Substep a (id=f4eec23b175fe854, index=3) returns an error.
...
[default]: 1 pending step: default_1

These behavior are as expected and are very nice.

So is -k good enough for your application? To summarize

By default failure of a substep will terminate the step, and the workflow after other running steps are completed.
keep-going means that we observe an error, we keep going until all possible (other) steps are completed. new steps will be started, but in the end we report error for the entire sos run -k.
ignore-error was not implemented, but the idea is that errors are more or less expected and the workflow should continue, and in the end we report success for the entire sos run, unless the ignored error caused some other steps to fail.

I am not sure how useful ignore error is, but in reality, especially for simulations, some percentage of substeps are expected to fail (e.g. with unrealistic simulation parameter). If a global ignore error is too risky, perhaps
[step: ignore_error=True]

could be used to indicate the case for particular steps.

So is -k good enough for your application?

-k is not good enough, as you can see 4 out of 5 of the output files for default_1 would have been valid if the workflow can keep going without worrying about failed substeps. As my first proposal in the post above outlines, I would expect to see no error from SoS and the output would be something like a1.txt, a3.txt.zapped, ... a5.txt , and additionally a warning providing some summary of failures. Currently with -k there are still errors and the workflow will stop.

By default failure of a substep will terminate the step

Really? I dont think this is the default behavor now, as you can see in the warning message in my post above, Terminating step after completing 3 submitted substeps.. That is why I argue in the 2nd proposal in my post above that it would be nice to have some way to terminate the step immediately as soon as one substep fails.

unless the ignored error caused some other steps to fail.

But I was talking about at substep level ignore. Imaging you have a very complicated workflow where the first step has 10K substeps, but only (randomly) 50 of them dont work, for various reasons. What I would hope to see is some way to run the workflow with the 2nd step taking those (10K - 50) not failed runs and continue, rather than having to require me fixing each of those 50 failures before being able to get any meaningful results. Of course in order to do so, I expect that we have to create dummy output files like .zapped to satisfy input / output dependencies.

I expect that we have to create dummy output files like .zapped to satisfy input / output dependencies.

zapped is for real file that was zapped, not for non-existing files. As I have said, keep going only keep going the rest of the workflow, and the step with failed substeps still fails and the entire workflow still fails.
What you are asking is a lot like the ignore error behavior where the errors are ignored. How to handle missing _output is a problem but I could prefer empty step_output over zapped placeholders. This will make the following step a bit more difficult but perhaps specifying
input: group_by=1

again could discard empty _input from the step_input so that you can process step_input normally?

What I was proposing is something like
[step_10: ignore_error]
input: for_each=dict(i=range(10))
output: f'test_{i}'.txt

fail_if(i==3, 'failed')
_output.touch()

[step_20]
input: group=1
# continue processing with reduced number of substeps

[step_30]
input: group='all'
print(f'Number of successful simulations are {len(step_input)}')

zapped is for real file that was zapped, not for non-existing files. As I have said, keep going only keep going the rest of the workflow, and the step with failed substeps still fails and the entire workflow still fails.

Sure, I understand the current behavior and I showed and agreed in my first post that current behavior is nice. I also only use zapped to refer to a dummy input/output file and a possible mechanism to make this behavior happen. We definitely can and want to use a different file extension for it.
Yes I wanted to discuss --ignore-errors  behavior and for my user case as you see, a global --ignore-errors behavior would be fine.

How to handle missing _output is a problem but I could prefer empty step_output over zapped placeholders.

sure! I was thinking of having some file extensions like zapped so the next step will know about it and ignore it by setting the corresponding output also to an empty file with certain extension.

again could discard empty _input from the step_input so that you can process step_input normally?

It would not work for my case (the way I implemented DSC benchmark workflow) because all files names are pre-determined and a step will explicitly look for all specified input names. So your proposal above would not work for that case.

So what you are saying is that in the case of
[step_10]
input: for_each=dict(i=range(10))
output: f'test_{i}'.txt

fail_if(i==3, 'failed')
_output.touch()

[step_20]
# continue processing with reduced number of substeps

[step_30]
input: group='all'
print(f'Number of successful simulations are {len(step_input)}')

step_20 will just skip the substep with empty _input because it is an error caused by the upstream step?

Yes exactly. But step_20 will explicitly require input and output, even though it decides it can process with reduced number of substeps based on number of failures occurred upstream. My toy example in the first post of this ticket suggests that behavior -- you see there I do expect to see complete list of files in output of default, not some dynamically determined subsets from successful runs.

OK, -i is available but many would consider it as --input. Not sure what else sounds like ignore but I would open to something like
-e keep-going
-e ignore

which is more readable than -k and -i.

I agree -e is reasonable. And just a reminder that we have not discussed the behavior to kill the current running step immediately when a substep fails. So perhaps we need:
-e keep-going
-e ignore
-e kill

?

-e kill will be useful for say if I have a typo in my script that will ruin all 10K runs ... we want it to terminal sooner than later. But alternatively I think we can live without using -e kill but instead of
WARNING: Substep a (id=c32cfc1964fed41a, index=3) returns an error. Terminating step after completing 3 submitted substeps.

we tell the users where to look for the error? eg, where stderr file is stored. That case users can examine it while other substeps still runs, and decide if to wait until later or to ctrl-c it immediately.

ok, they are on my radar now, will be done after -r host with templates.

Great -- i want to bring up a point that might matter: sometimes a failed substep is not necessarily identified by a non-zero return code. It could be, say out of walltime or memory, or simply randomly killed on the cluster. So maybe we cannot merely rely on the conventional error handling mechanism (eg, try ... catch) to creating empty output files.

@BoPeng no rush for the next couple of days, but I'm adding a soon label to this ticket if it can be prioritized. Currently there are a few of my simulation runs that simply cannot complete due to resource limitations. But failure to complete them on the cluster currently disrupts workflow that follow those failed steps. I think -e ignore is the only solution to the problem.

Yes. It is hard to work on this in Denver with ski on. :-)

-e abort is missing. -e ignore is being implemented for various cases.
import time

[10]
input: for_each=dict(i=range(10)), concurrent=False
output: f'test_{i}.txt'

_output.touch()

fail_if(_index == 5, 'fail at 5')

[20]
output: _input.with_suffix('.bak')
_output.touch()

is the test case for failure in one of the substeps.

@BoPeng here is a test for -e ignore with dynamic input -- that is, previously failed step will not generate some inputs the next step is expecting. The desired behavior is to be able to ignore all these problems with a summary of ignored errors at the very end.
issue_1266.tar.gz
To reproduce, simply:
sos run test_run.sos

https://vatlab.github.io/sos-docs/doc/user_guide/error_handling.html tries to document the new features.
It appears that keep-going could try to execute the next step, but then it would be the same as ignore just with different final return code of the workflow.

@gaow Frankly I do not know the intended behavior for subworkflows. In your example,

simulate is called but only generates 4 out of 5 needed targets
a_simulate is expecting 5 from sos_run('simulate'). It only got 4, so a_simulate is reported to be failed.
The failure of a_simulate makes sos_step('a_simulate') unavailable, so a_test cannot continue. The same goes to default.

until you have a clear logic as what error should be ignored at the step level

Not sure if this "logic" is clear --- I need a step to success and generate all the output files that are explicitly required by the next step using dynamic. So

a_simulate should not return failed status.
a_test should run even some of its input are not available because of failed output from previous step.

That was why I thought making some fake input like zapped (with a different name) can help.

I do not know the intended behavior for subworkflows

Well, perhaps subworkflow is not the best way to do it, but the idea (or the goal) is to have prespecfied input and output file names. My workflow steps are supposed to generate them. If some of those files are not generated, I am hoping they can be replaced with dummy files to keep the process going. I'm definitely open to other possibilities to achieve it. I'll DM you for additional details.

Behavior of nested workflow is improved, but I am not sure if the behavior is optimal.
(sos) [bpeng1@BCBMF5KWK0F8F69:~/sos/sos/issue (master *)]$ sos run test_run.sos -e ignore -s force
INFO: Running simulate in pipeline #1:
INFO: simulate in pipeline #1 is completed (pending nested workflow).
INFO: Running module simulate:
INFO: simulate (index=0) is completed.
INFO: simulate (index=1) is completed.
INFO: simulate (index=2) is completed.
INFO: simulate (index=3) is completed.
WARNING: Ignoring error from substep simulate (id=b7af26066c5f1731, index=4):
Failed to execute Rscript .sos/simulate_4_5ddd04ef.R
exitcode=1, workdir=/Users/bpeng1/sos/sos/issue, stderr=test/simulate/simulate_5.stderr
---------------------------------------------------------------------------.
INFO: simulate output:   test/simulate/simulate_1.rds test/simulate/simulate_2.rds... (5 items in 5 groups)
INFO: a_simulate output:   test/simulate/simulate_1.rds test/simulate/simulate_2.rds... (5 items)
WARNING: Output target test/simulate/simulate_5.rds does not exist after the completion of step a_simulate
INFO: Running test in pipeline #1:
INFO: test in pipeline #1 is completed (pending nested workflow).
INFO: Running module test:
WARNING: Failed to regenerate or resolve test/simulate/simulate_5.rds.
WARNING: Error from step test is ignored: [test]: Target unavailable: test/simulate/simulate_5.rds
INFO: a_test output:   test/test/simulate_1_test_1.rds test/test/simulate_2_test_1.rds... (5 items)
INFO: Running default:
INFO: default is completed.
INFO: default output:   test/test/simulate_1_test_1.rds test/test/simulate_2_test_1.rds... (5 items)
INFO: Workflow default (ID=0825ced33264897e) is executed successfully with 4 completed steps and 8 completed substeps.

Well, the workflow seemingly completes but I expected to see 4 outputs in both simulate and test folder under test. Currently,
$ find . -name "*.rds"
./simulate/simulate_4.rds
./simulate/simulate_2.rds
./simulate/simulate_3.rds
./simulate/simulate_1.rds

only has results for simulate. simulate_5.rds is missing. The desired behavior, if possible, is to let the test step still work on the 4 simulate_*.rds and generate 4 test_*.rds.

This is because the test step cannot run at all due to missing one of the input files. -e ignore can handle missing output but not missing input because none of the substeps can start.

Right ... so the question now is how to achieve it one way or another -- including the option of changing the way the workflow is written if that helps (as explained in the slides I sent you earlier this evening).

One way is to proceed, in the -e ignore mode, missing files are not checked initially. Instead it will be checked after group_by, and we proceed with substeps with valid _input.

#1329 allows test to run because now
input: dynamic(test_input_files) , group_by = 1

now automatically removes files that do not exist.

One way is to proceed, in the -e ignore mode, missing files are not checked initially. Instead it will be checked after group_by, and we proceed with substeps with valid _input.

This is implemented so the workflow would execute the four groups even without the dynamic decorator.

Thank you @BoPeng now the behavior is as expected. However, can we simplify and rearrange the warning message a bit? That is, the text:
Failed to execute Rscript .sos/simulate_4_93141e5e.R
exitcode=1, workdir=/home/gaow/tmp/02-Dec-2019, stderr=test/simulate/simulate_5.stderr
---------------------------------------------------------------------------.

is best saved and shown at the very end before the entire workflow quits, so they dont hide somewhere in the middle of very long screen output that users cannot locate.
Also shall we consolidate this line to the previous warning? eg,
WARNING: Ignoring error from simulate (id=b7af26066c5f1731, index=4, output=test/simulate/simulate_5.rds):

@gaow Our interpretation of keep-going and ignore-errors is different from GNU Make (https://stackoverflow.com/questions/53039550/makefile-ignore-errors-vs-keep-going). I think,

what we are doing now with ignore-error is the keep-going of GNU make.
ignore-error should be something different, name ignore non-workflow-level errors such as when an action fails.

Therefore we should perhaps merge the behavior of ignore-error to keep-going and limit the use of ignore-error in a smaller scale. However, GNU Make allows the combined use of -i -k and we cannot, so things get tricky here.
I guess the problem is that I do not understand why keep-going does not proceed like ignore-error in the example of https://vatlab.github.io/sos-docs/doc/user_guide/error_handling.html  .

I would propose the removal of -e keep-going since it does not provide any additional feature than -e ignore.

Indeed ignore-errors simply means ignore non-zero return code from a system call, it seems. But if the output still dont exist, it will fail unless we also let it keep-going.
It has happened to me that for some crazy C++ programs someone wrote they decide to return 1 instead of return 0 for the main function ... I had to write a wrapper around it for SoS to work. With ignore-errors it might help. (or we have a better name for it?)

Therefore we should perhaps merge the behavior of ignore-error to keep-going and limit the use of ignore-error in a smaller scale.

I think you are saying:

-i -k in GNU Make = -e keep-going in SoS
-i in GNU Make = -e ignore-errors in SoS

right?

-i -k in GNU Make = -e keep-going in SoS
-i in GNU Make = -e ignore-errors in SoS

Currently:

With -e keep-going we stop a brach at the step where an error happens, and we execute "safe" branches, and stops after all branches are executed or failed. Everything should be safe because all errors are caught.

With -e ignore, we ignore the fact that a step has failed and venture to the next step. which is different from the -i of GNU Make, and is like -i -k. The problem with -e ignore is that it can be unsafe to execute after a failed step since the step 20 will have different results with/without error in step 10.

[20]
input: dynamic('*.txt')

So in summary

We execute more and more in the order of abort, default, keep-going, ignore.
ignore is not -i in GNU Make. more like -i -k.
We could implement GNU Make like -i to only ignore errors from actions and scripts, not from workflow execution, but we will need another name for -i -k.

I agree with your summary. For 3) we can call it ignore-exception ? for current ignore how about skip? That is, skip errors.

I do not want so many options but sometimes fine-controls are needed.

For actions we do have allow_error, although

sh: allow_error=True
  rm somefile

is a lot more trouble some than
    -rm something

in GNU make.

keep-going has a right balance of safety and gentle stopping, because all errors are caught. However, I feel that it is not a useful feature because the user would need to rerun the workflow anyway. I can even suggest that **we make keep-going the default so that SoS would by default execute all executable steps safely. They can do -e abort is they want a harsher stop.

Then we need to figure out what we are "ignoring" right now, and if we should separate into finer options.

we make keep-going the default

Great idea! We then also dont confuse it with the option in GNU Make

For actions we do have allow_error

Sorry I keep forgetting that. This is good enough I think.
To recap we'll make keep-going the default, then have abort and ignore? The question is whether it is worth keeping both keep-going and current default.

The question is whether it is worth keeping both keep-going and current default.

The current default should be removed.

Agreed. Then we dont have either keep-going or ignore-errors to confuse with Make. Although maybe we can still consider using the word skip over ignore

I am glad that there is one mode to remember, especially for less confusion for differences between default and keep-going and between keep-going and ignore.
Pushed and https://vatlab.github.io/sos-docs/doc/user_guide/error_handling.html is updated. The document will be updated with more example for how -e ignore ignores different types of errors.

Nice! Before you update examples, how about the behavior of error messages, with user-friendliness (concise and easy to track) in large benchmarks in mind (https://github.com/vatlab/sos/issues/1266#issuecomment-561226798)?

Currently I see four warnings
WARNING: Ignoring error from substep simulate (id=b7af26066c5f1731, index=4):
Failed to execute Rscript .sos/simulate_4_8cc2f7f1.R
exitcode=1, workdir=/Users/bpeng1/sos/sos/issue, stderr=test/simulate/simulate_5.stderr
---------------------------------------------------------------------------.

WARNING: Output target test/simulate/simulate_5.rds does not exist after the completion of step a_simulate

WARNING: Output target test/test/simulate_5_test_1.rds does not exist after the completion of step a_test

WARNING: Output target test/test/simulate_5_test_1.rds does not exist after the completion of step default

The last two complained about the same missing file, but for different steps. The first warning message is the real message, but it is pretty difficult to move to the end because sos will need to accumulate all four messages and display them at the end, and then it makes it difficult to figure out when the errors happened if we move them out of the location they were triggered.. I would suggest a separate enhancement ticket for it.

it is pretty difficult to move to the end because sos will need to accumulate all four messages and display them at the end,

I can imaging that ...
but if we could remove the warming about missing files and put the file name into the first warning, then at least when people track for warning messages they will scroll up and get right to the real warning. Now with all 3 messages people only have 1/3 chance looking at the right one ...
WARNING: Ignoring error from simulate (id=b7af26066c5f1731, index=4, output=test/simulate/simulate_5.rds):

(see how output=test/simulate/simulate_5.rds is added to the message above)

remove the warming about missing files '

No. We do not know what file(s) the script will generate, and if the failure will cause the missing of them.

and put the file name into the first warning,

These look like legitimate warning messages because the steps and workflows got an error, although in this case they are nested workflows with the same missing output file.

The confusing part could be
INFO: a_test output:   test/test/simulate_1_test_1.rds test/test/simulate_2_test_1.rds... (5 items)
WARNING: Output target test/test/simulate_5_test_1.rds does not exist after the completion of step a_test

You see the INFO says 5 files (in green) and then one missing. In theory they can be combined as
..... (5 items, 1 missing).

I do not particularly like https://github.com/vatlab/sos/commit/d8fee925b5acc980ed31f9fded1ac00865cbbfbc but we can keep it if you prefer having one message instead of two.

I think it reads good -- I made a couple of commits to further improve it, based on what I see running sos run test_run.sos from this ticket.
So we are all set now? This is very nice.

@BoPeng additional concern: when for task executions on cluster a task died silently due to wall time or memory limit on SoS level it receives a message along the lines of "task is inactive for ?? secs". In that scenario will what we've done here still help? That is with -e ignore  i hope to be able to skip the substeps that takes long time or lots of memory to run. That was what motivated me to bump this ticket. I can certainly test and see if we expect it to work as of now.

That is another ticket that can only be debugged on your cluster (mine has a 1 hr limit on walltime). The failed tasks will be ignored with -e ignore so your workflow should complete though.

@gaow I remember that you posted an example with 2 minutes jobs etc, but I cannot find it now. Could you send me a link?

@BoPeng sure this is another opened ticket:
https://github.com/vatlab/sos/issues/1323#issuecomment-557846407

I dont have a reproducible example now, but although -e ignore option skips failed runs the workflow hangs without ultimately completing. I'm on the cluster so maybe there is still communication issues with job scheduler when errors are ignored? I'll try to come up with MWE later but just in case there are something obvious to check now.

I saw it in one case and fixed it, but I could not get a simple enough example to be used as a testcase. Basically ignoring error without proper signaling will cause another step to wait for something. I found the problem by comparing sos_debug.log generated without error and with error and -e ignore.

Hmm I'm afraid there is a bigger problem here. My benchmark pipeline was completely broken after the update -- my script simply now takes in the wrong dynamic input files. I'm still investigating (by rerunning without -e ignore) if the new feature cause the issue, but what's the behavior for this:
input: dynamic(mnm_high_het_input_files) , group_by = 3,

suppose I've got ABC, and DEF. If B fails and was ignored, is it going to regroup as ACD and EF?

If B fails and was ignored, is it going to regroup as ACD and EF?

Yes. This is #1329.

If B fails and was ignored, is it going to regroup as ACD and EF?

Note that without dynamic, the entire ABC would be ignored due to missing B.

I see. this is the problem ... because it breaks the logic in this workflow. I'd appreciate suggestions here (including revisiting the way test_run.sos is generated) ... For now I'm going to rerun without -e ignore (and with sos_debug.log) to verify that other recent changes don't break my workflow.

But why do you have to use dynamic? It is designed for dynamic('*.txt'), not dynamic('output.txt').

Hmm you are saying we dont need dynamic() in test_run.sos? I was pretty sure at some point I tried without dynamic but it complains that input do not exist. Maybe since I also use depends: sos_step() then input don't get checked before they are actually generated by the previous step and that makes it unnecessary to use dynamic?

Can you try without dynamic? If you are sure the input should exist when the step is executed, then it should be arranged to be executed after other steps and dynamic is not needed. Here you use dynamic as a convenient way to stop SoS from running it as "head" of a branch, and we should discuss the best way to do it as a separate topic.

I'm trying without dynamic and also without -e ignore things seem to work. I've pushed revised version here:
https://github.com/gaow/random-nbs/blob/master/slides/20191201_DSC_SoS/sos/test_run.sos

If you are sure the input should exist when the step is executed,

Well with depends: sos_step all inputs should have been generated before a step actually executes. Previously I used dynamc probably because I did not have depend on steps back then, or for whatever other reasons it compains about missing input that I dont recall.

we should discuss the best way to do it as a separate topic.

Will do.

Some examples crafted at https://github.com/vatlab/sos/issues/1329#issuecomment-564097025

I tried with -e ignore again. My workflow completed properly with a few corner cases identified that led to failure. After fixing those corner cases and running again SoS correctly skipped completed runs, with tasks regrouped focusing on those previously failed corner cases. It then completed. I cannot reproduce the hang behavior that was caused by messed up grouping due to the use of dynamic (this new run I stopped using dynamic)

OK. Thanks for your update. I think this would alleviate the performance problem you mentioned in #1321 . Actually, after some more thought,
task:
input:

does not achieve much since currently master tasks only saves one copy of the script anyway. The only difference is that with
task:
input:

we are resolving substeps when the tasks are executed, and with
input:
task: trunk_size=0

we are resolving substeps for each subtask. The latter has the advantage of skipping completed substeps.
I will close this ticket for now.


RangeError in /clan/gente-latina-pjr0qj92

Error in ClashLeaders
RangeError in /clan/gente-latina-pjr0qj92
Maximum call stack size exceeded
View on Bugsnag
Stacktrace
webpack:///node_modules/vue/dist/vue.esm.js:891 - ke.walk

View full stacktrace
Created automatically via Bugsnag

An error linked to this issue has been marked as fixed in Bugsnag
RangeError in /clan/gente-latina-pjr0qj92

An error linked to this issue has been marked as fixed in Bugsnag
RangeError in /clan/hellsing-jq028p8


Docker support

This is the second time I’ve seen this repo posted, so I thought I’d ask about docker support 😄 There used to be https://github.com/lambci/lambci, but it is no longer maintained
Docker images can be found on https://hub.docker.com/r/amazon/aws-lambda-nodejs and https://gallery.ecr.aws/lambda/nodejs
More information on using docker with AWS Lambda can be found:

https://docs.aws.amazon.com/lambda/latest/dg/nodejs-image.html
https://docs.aws.amazon.com/lambda/latest/dg/images-create.html
https://www.serverless.com/blog/container-support-for-lambda

These resources may also be helpful:

https://www.npmjs.com/package/aws-lambda-ric
https://docs.docker.com/develop/develop-images/multistage-build/

@jlarmstrongiv Thank you for raising it. To clarify what the ask is: are you suggesting everynode provides base docker images for every version of node.js in addition to layers like it does today?

@tjanczuk yes, that’s correct.
I’ve used a mix of regular serverless functions and docker serverless functions for both professional and personal projects since they were released a few years ago.  There’s always some troublesome dependency or >50mb function that needs it.  As I share code between both, it’s important that they all run on the same version of node.  In a perfect world, everynode.
Having the docker images built on the ci regularly like lambda layers would be perfect.


Support initial timestamp in shard iterator

Also allow context to pass through to aws sdk so we can cancel stalled requests.  Kenesalite currently hangs on AT_TIMESTAMP requests when it has no work (see https://github.com/mhart/kinesalite/issues/84).
The reason for this change (and the fork) is to be able to poll kinesis at intervals, which is more energy efficient than polling it constantly.

I'm not sure about submitting upstream.  The context change seems sort of fundamental and inconsistent (since i'm not doing it all places).


"as matrix" error in explore_features function.

Users in the boot camp is facing "as_matrix" error when calling ‘ai.explore_features(raw_data)’ in the AI Workbook.

Issue is because of recent changes of pandas and yellowbrick library changes. Yellowbrick library is not getting upgraded to the latest version.
The issue is fixed and the latest version 1.0.2. is released.


Limit freeze dry versions. Closes #42

Main things to do:

[x] User can define in the settings that Freeze Dry Versions are made for each page they bookmark via the browser. For that we have to listen to the bookmark event.
[x] They can "Archive Current Page" via a button in the Dropdown, not the same as bookmarks though
[ ] We automatically store the last 100 pages as freeze dry, and if they are not bookmarked or manually archived, we delete the freeze dry version.

Part 1 is simple. UI done for a checkbox in settings to allow user to set a flag in local storage. Implementation of this being done in #46
Part 2 will involve a new button in popup. The button action should set a flag, say keepFreezeDry, on the page doc associated with the current page*. This flag will be used in part 3. Harder part is keeping the state of that button set (it's essentially a toggle); popup is in plain JS + HTML.
* There is no guaranteed way to get the current page. It will have to do a query on page docs with the same URL and take the latest, I guess.
Part 3 is by far the most difficult part. I'm currently thinking a scheduled task every hour (?) to remove freeze-dry attachments from all page docs that are not:

the latest 100 page docs
pages with keepFreezeDry flag set
pages with associated bookmark doc (IF option from part 1 is set)

Will look into possibility of scheduled task in background script (maybe this if it works in browser). Other option is to do it on the page visit event, but the queries involved will add more overhead to a page load, which we don't want.

the latest 100 page docs

Idea: make a list of (1 to 100+n):url-key-pairs in localstorage, every 30(?) minutes, check for all that have the key 1 to (100+n-100) without a the keepFreezeDry-flag, then rewrite the localstorage list with the remaining n, so that they have the keys 1 to n.

node-cron package seems to work quite nicely. This can be used to schedule freeze-dry cleanup in the background script. Every 30 mins sounds alright.
@oliversauter Not sure if I'm fully understanding your algorithm here, but it doesn't seem to add anything that can't be done already with MongoDB-style queries via pouchdb-find. The latest 100 docs bit is simple, as the docs are sorted by _id and thus timestamp, hence can just be skipped in the query (there's a skip parameter which skips specified amount of docs). The biggest difficulty of the query is:

pages with associated bookmark doc

as it requires a second query on bookmarks to see if they are associated with any of the page docs with freeze dry. This adds a lot of unwanted complexity, but could be mitigated by adding a flag on page docs generated on bookmark create event. @arpitgogia what do you think about this for #46?
Another little hiccup is that the query should check for existence of the freeze-dry on page docs. No need to return page docs that don't have freeze-dries attached. This is easy in pouchdb-find using the $exists operator, however our freeze-dries are attached to page docs using the name frozen-page.html. MongoDB/PouchDB-find use the period/dot character as a special character to indicate nested fields, so the existence query does not work. Put an issue upstream to see if there are any work-arounds for this, or if the dot character is just not supported in field names (if so, I will rename our freeze-dry attachments).

Wrote the basic cleanup logic with a schedule for 30mins via background script. Doing the attachment existence check in JS now, due to the problem mentioned in prev comment. This is fine, and produces the same result, however it should be very wasteful as the DB query will load unnecessary docs into memory whether or not they contain the freeze-dry attachment. To be optimised depending on if there's an easy work-around for the $exists operation, or via renaming the freeze-dry attachment.
RE the popup archive button:
Mostly fine, but as there isn't a guaranteed 1-1 association between the current browser tab and a page doc in pouch, it may be disabled on some pages. Default state is disabled, but should become available pretty quickly in most cases. How it works:

on click of popup button (onload event), latest page doc with same URL as current tab is attempted to be found
if not found, leave the button disabled
if found, but the freeze-dry flag already set, leave disabled
if found, set the page doc ID as a data attribute on the button and enable the button

Once pressed, it should update the freeze-dry flag on the associated page doc, and given the onload conditions, the next time the user open the popup on the same page, the button should remain disabled (as they already set it for this page).

if not found, leave the button disabled

Why not put a button "Archive page" if not found?

Why not put a button "Archive page" if not found?

If it's not found it's usually because the url is blacklisted, special page (browser settings page) or, most commonly, the page doc is yet to be inserted (user opens popup as soon as loading the page). Everything happens asynchronously so better to think of the button being disabled as a loading state, as it will get enabled when ready.
Set up a background scheduled-tasks module which can be used for any type of recurring background script service. Also a small util library wrapping around the browser idle API (thanks to @oliversauter for making me aware of this), so any function, async or sync, can be augmented to only run after a user is deemed idle, which will be useful for other tasks.
Scheduled task set for the freeze dry cleanup which will only run when the user is deemed idle, and makes sure that it only happens once every 30 minutes. Indexes set up as well so it's nice and speedy. Tested it quite a bit this morning, and was working as intended, but will continue to monitor as I use the browser to see how it handles things.

Great that that worked out so quickly with the idle mode :)
Sweet!
Regarding the loading state of the button, how about we have a loading spinner there, indicating that it is loading? Or how long does it take for the loading to happen usually?

Usually takes a couple of seconds. It's however long the background script's activity-logger takes to create a page stub for the current tab, which all happens asynchronously.
I thought about having three button states - enabled, disabled, and loading - but the problem is that there is only two available states to work with: either page is in DB or it isn't. To support that third state would require the popup script to talk to the background script, similar to how imports works. While doable, I think is a lot of extra complication for small gain right now. Maybe we can see how it feels from UX perspective now with two states and introduce the third if it feels awkward/confusing?
Main disadvantage with two states is that you don't know if it's going to become available or not. I think most use-cases would see the user reading through a page for a little bit first before choosing to archive. It may be confusing for the use-case where you're going to a bunch of pages with the intention of immediately archiving them.


aggregated table is missing one of the google search terms (columns)

missing 'search_trends_generalized_tonic–clonic_seizure'
There are 422 columns in the google table, but only 421 in the aggregated table.

Thanks for reporting this! It should be fixed via https://github.com/GoogleCloudPlatform/covid-19-open-data/commit/36a125cc2d0e4220c3e5158b302d653157dc9ea2 (we remove all dashes, but that symptom name has a special character dash which we didn't account for)


Example/cumulated calls trees

Coverage increased (+0.009%) to 99.27% when pulling b187e6e41cb0890e63e708c380555840660575a1 on example/CumulatedCallsTrees into 996a6fcb55bebd9c205a6c4a78602653166828a9 on master.

Coverage decreased (-0.004%) to 99.257% when pulling 6a9eb7e95f0a5accd09530d3785189575aba3dc6 on example/CumulatedCallsTrees into 996a6fcb55bebd9c205a6c4a78602653166828a9 on master.

Coverage decreased (-0.004%) to 99.257% when pulling 6a9eb7e95f0a5accd09530d3785189575aba3dc6 on example/CumulatedCallsTrees into 996a6fcb55bebd9c205a6c4a78602653166828a9 on master.


US EOY epic reminder

Adds a special reminder to the epic for the US end of year campaign.
This feature automatically finishes on 27/12/20.
The reminder email will be sent on 28/12/20.
CTA

Clicked

Payload

Confirmation

@tomrf1 For consistency, shall we remove "in" from the header in the pre-contrib reminder?


why always "Build failed with 62 weighted issues."

my project:
classpath "io.gitlab.arturbosch.detekt:detekt-gradle-plugin:1.14.1"
my module:
apply plugin: 'io.gitlab.arturbosch.detekt'
detekt {
failFast = false
buildUponDefaultConfig = false
}
tasks.detekt.jvmTarget = "1.8"
or
detekt {
failFast = true
buildUponDefaultConfig = false
}
my detekt task still failed like this:
Execution failed for task ':vivospace_forum:detekt'.

Build failed with 62 weighted issues.

whatever i set failFast true or false
every time deteke failed .
but i want detekt just report issues ,not cause build failed

You will need to create a config file and add this:
# config/detekt/detekt.yml
build:
  maxIssues: -1

Should we add a flag in the gradle plugin to overwrite this directly... something like ignoreFailures in ktlint-gradle?

In addition to Brais comment I'd like to give more information by mentioning this link.
https://detekt.github.io/detekt/failonbuild.html

Should we add a flag in the gradle plugin to overwrite this directly... something like ignoreFailures in ktlint-gradle?

I wouldn't do it, since it was a clear decision which settings to move into the config section and which into the CLI or Gradle plugin. Perhaps this creates even more confusion than it brings to the table at the end of the day.

We happen to already have an ignoreFailure flag:
detekt {
    ignoreFailures = false                                // If set to `true` the build does not fail when the maxIssues count was reached. Defaults to `false`.

From https://detekt.github.io/detekt/gradle.html