A couple patches

Never done a pull request before.  Hopefully this works...

Hey jjcf89! Thanks a ton for the contribution, I definitely wasn't expecting that, I'm blown away!
Thanks for integrating cross-platform support, that's definitely something I'll use in the future, as I wasn't aware os.system('cls') was Windows specific. And also thanks for fixing those errors and increasing readability, that's awesome!
I'm curious about the gitignore file, I Googled around a little but I couldn't myself find any use for it, how does it assist you, if I may ask?

The .gitignore file tells git which files it doesn't need to worry about.  For example if when I open a file and a .py~ file is created and I don't want git to list it as a untracked file I can create a rule to ignore those files.
https://git-scm.com/docs/gitignore
There is actually a github repo that includes indepth .gitignore files for different languages. And I believe github itself will create you .gitignore if you tell it what language you are using when you create it.
https://github.com/github/gitignore

I see, I understand now , thanks a lot! It's a bit curious however, it seems that each individual developer may have their own particular needs, and the gitignore file doesn't seem to ignore itself, so for example, I've added demo.gif to it because I had the gif I posted on reddit in the same folder and I wanted Git to ignore it, but now, if you were to pull the latest version of the master branch, you'd have demo.gif on there too, which seems a bit strange.
Anyway, thanks again! :+1:

Yeah its not perfect but it doesn't really hurt to ignore things that other people won't have anyways.
On bigger projects, it can happen that someone will add a ignore that unintentionally hides some file they wanted to check in and forgot about.  But it doesn't happen that much.


Composer command conflicts with PHP composer (Dependency Manager)

Some devs have experienced the command conflict when they have already set up Laravel, etc. PHP dev enviornments in their machines.
Possible solutions:

Change command to bcompoer or something meaningfull
Document this in the troubleshooting section under composer docs

+1 to this. I ran into the same thing when trying to install to evaluate Ballerina.

Duplicate issue https://github.com/ballerina-platform/ballerina-lang/issues/9442

Closing this issue and please refer to https://github.com/ballerina-platform/ballerina-lang/issues/9442 for further tracking.


remove command returning "Error: the project specified does not exist"

Testing with the commands to resolve other issues, I've found out if you add members to a standup you cant remove them

#75


Add Size Leaks

Size Leaks are very important XS-Leaks. They are often addressed in the browser when researchers find exact ways to leak the size. Some mechanisms that still work:

Cache Timing (Original Ref) can be used to predict the size of a request.

Fixed, but relevant works (we should document them):

Flash Size Leaks. Flash is going to be removed by the end of the year Ref1, Ref2 (currently is under strict user permissions)
Attackers could abuse the way browsers use cache limits (Ref1 & Ref2) to predict the size of a resource using Cache API (Ref1 & Ref2). Browsers fixed this issue by adding random noise (Ref1-Chrome & Ref2-Firefox)
Video and Audio Parsing. This also involves CORB and CORP.
HEIST. I believe the fix was the same as the one in cache limits Ref1 & Ref2)

@manuelvsousa did "HEIST" get fixed it seems like window navigation should bypass "SameSite cookies" from https://github.com/w3c/resource-timing/issues/64#issuecomment-242785022
Considering the SharedArrayBuffer can be used to create a high precision clock https://github.com/whatwg/storage/issues/31 may not be enough to prevent this. https://xsleaks.dev/docs/attacks/timing-attacks/network-timing/#unload-events

Created https://github.com/xsleaks/wiki/pull/114 for HEIST


Using csharp alias for Markdown code triggers OmniSharp error checks

I'm writing README.md using GitHub version of Markdown - within a C# project - so OmniSharp is running, and as soon as I use csharp alias for code the OmniSharp triggers error checking against markdown document:

This requires me to restart Atom to get rid of these errors.
Can I somehow help with this?
v.0.23.0
Atom: 1.0.3
Thanks!

Is this project available anywhere to look at?  That's really weird... it sounds like it might be related to the grammar somehow, but I'm not 100% sure on that.

using generator:
yo aspnet

     _-----_
    |       |    .--------------------------.
    |--(o)--|    |      Welcome to the      |
   `---------´   |   marvellous ASP.NET 5   |
    ( _´U`_ )    |        generator!        |
    /___A___\    '--------------------------'
     |  ~  |
   __'.___.'__
 ´   `  |° ´ Y `

? What type of application do you want to create? Empty Application
? What's the name of your ASP.NET application? EmptyApplication
   create EmptyApplication/.gitignore
   ...
➜  development  cd EmptyApplication
➜  EmptyApplication  ls
Startup.cs   hosting.ini  project.json wwwroot
➜  EmptyApplication  atom .
➜  EmptyApplication  touch README.md

README.md
# README

Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.

## Description

```csharp

It's enough that I type csharp and press return key, it trigger syntax detection for C#:

updated to 0.23.1

I'm seeing it now.... it has to do with the linter and it trying to (for
whatever reason) lint the code inside the csharp block of the markdown...
Thats cool.... confusing, but cool.
I'll work on a fix for it, thanks for the extra steps!
2015-08-10 16:48 GMT-04:00 Peter Blazejewicz<EMAIL_ADDRESS>
using generator:
yo aspnet
 _-----_
|       |    .--------------------------.
|--(o)--|    |      Welcome to the      |

---------´   |   marvellous ASP.NET 5   |     ( _´U_ )    |        generator!        |
/A\    '--------------------------'
|  ~  |
'..'_
´    |° ´ Y
? What type of application do you want to create? Empty Application
? What's the name of your ASP.NET application? EmptyApplication
create EmptyApplication/.gitignore
...
➜  development  cd EmptyApplication
➜  EmptyApplication  ls
Startup.cs   hosting.ini  project.json wwwroot
➜  EmptyApplication  atom .
➜  EmptyApplication  touch README.md
README.md
README
Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum.
Description

It's enough that I type csharp and press return key, it trigger syntax
detection for C#:

[image: csharp]
<https://cloud.githubusercontent.com/assets/14539/9183769/9fbf31ba-3fb1-11e5-9e20-0d7882262767.gif>

updated to 0.23.1

—
Reply to this email directly or view it on GitHub
<https://github.com/OmniSharp/omnisharp-atom/issues/459#issuecomment-129602246>
.

@david-driscoll
Tested with 0.23.2 - fixed.
:bow:
Thanks!


New dependency tracking problem rises

As most automatic dependency tracking problems has vanished there is more to come:
[   47s] Finding  Requires: /usr/lib/rpm/nodejs.req
[   47s] error: Dependency tokens must begin with alpha-numeric, '_' or '/': >=>= 1.5.0
[   47s] npm(nomnom) < 1.6.0
[   47s] npm(JSV) >=>= 4.0.0
[   47s] npm(JSV) < 4.1.0
[   47s] nodejs(engine) >= 0.8.0
[   47s]
[   47s] error: Failed to find Requires:
[   47s] Provides: jsonlint = 1.6.2-9.1 npm(jsonlint) = 1.6.2 own(ansi-styles) = 1.1.0 own(has-color) = 0.1.7 own(strip-ansi) = 0.1.1 own(chalk) = 0.4.0 own(underscore) = 1.6.0 own(nomnom) = 1.8.1
[   47s] Requires(rpmlib): rpmlib(PayloadFilesHavePrefix) <= 4.0-1 rpmlib(CompressedFileNames) <= 3.0.4-1
[   47s] Requires: npm(ansi-styles) >= 1.0.0 npm(ansi-styles) < 1.1.0 npm(nomnom)
[   47s]
[   47s]
[   47s] RPM build errors:
[   47s]     Dependency tokens must begin with alpha-numeric, '_' or '/': >=>= 1.5.0
[   47s] npm(nomnom) < 1.6.0
[   47s] npm(JSV) >=>= 4.0.0
[   47s] npm(JSV) < 4.1.0
[   47s] nodejs(engine) >= 0.8.0
[   47s]
[   47s]     Failed to find Requires:
[   47s]
[   47s] wildcard3 failed "build jsonlint.spec" at Wed Sep  6 13:20:31 UTC 2017.

from https://build.opensuse.org/package/show/home:illuusio:nodejs-bundle/jsonlint
and
[   27s] error: Dependency tokens must begin with alpha-numeric, '_' or '/': >=. 0.0
[   27s] npm(mime-db) = 1.24.0
[   27s] npm(mime-db) <. 0.0
[   27s] npm(mime-db) >= 2.0.0
[   27s] npm(mime-db) < 3.0.0
[   27s] npm(statuses) >=. 0.0
[   27s] npm(statuses) = 1.3.1
[   27s] npm(statuses) <. 0.0
[   27s] npm(statuses) >= 2.0.0
[   27s] npm(statuses) < 3.0.0
[   27s] nodejs(engine) >= 0.8.0

from https://build.opensuse.org/package/show/home:illuusio:nodejs-bundle/hexo-server

Never ending swamp.. it seems..

well...that's nodejs...I'll check it again :-(

@illuusio can you please tell me what '>= 1.5.x' will expand? I'm a little confused...
because literally 1.5.x itself means '>=1.5.0 <1.6.0', but what is '>= 1.5.x'?

@illuusio can you please tell me what '>= 1.5.x' will expand? I'm a little confused...
because literally 1.5.x itself means '>=1.5.0 <1.6.0', but what is '>= 1.5.x'?

Kind of hard question but I say everything above 1.5 but under 1.6 is correct (just forget last .x because it ain't something we need to take care of).
It took a while to seek but I think it would need to be done this way: https://serverfault.com/questions/411444/rpm-set-required-somepackage-0-5-0-and-somepackage-0-6-0/411491.
So for rpm version 1.5.x would be 'somepackage >= 1.5, somepackage < 1.6'.

@illuusio actually it was the correct behavior...eg ">=(whitespace here)1.0.0", should not be interpreted as '>= & 1.0.0' => ‘>=.0.0 & 1.0.0', but should be interpreted as '>=1.0.0'. so '>= 1.5.x' is '>=1.5.x' and then 1.5.x is [>=1.5.0, <1.6.0], then [>=>= 1.5.0, <1.6.0] comes...maybe if:

1.0.x version or 1.x.x version
just two elements eg '>=' and '1.5.x'
then '>=' should be filled to '>=0.0.0'

then it'll be '>=0.0.0 >=1.5.0 < 1.6.0', and finally generates [>=1.5.0, <1.6.0]

@marguerite Yes it seems correct but again '>=1.5,<1,6' is also correct but honestly '>=1.5.0,<1.6.0' is bullet proof. Nodejs 'flexible' version scheme is also painful to adapt.

Ok then there can be depency that is available as own() but not npm() like in 'https://build.opensuse.org/package/binary/home:illuusio:nodejs-bundle/js-beautify?arch=x86_64&filename=js-beautify-1.6.3-9.2.noarch.rpm&repository=openSUSE_Leap_42.3' where I needed to build config-chain to satisfy depencies but it's also with rpm package but marked as 'own(config-chain)' and package requires 'npm(config-chain)'

oh I see, this is a wrong behavior.

I think this have been solve in PR so I close this


Build broken? StackOverflowException on BuildLib (macOS)

Hey Fable team,
I've tried to build this project and I can't get it working somehow:
Starting Target: BuildLib (==> CopyModules)
/usr/local/bin/npm run build-lib

><EMAIL_ADDRESS>build-lib /Users/bo/src/fable-repl
> fable-splitter -c src/Lib/splitter.config.js

fable-splitter 2.1.8
fable: Compilation started at 7:03:12 PM
fable-compiler 2.3.6
/Users/bo/src/fable-repl/src/Lib> dotnet restore Fable.Repl.Lib.fsproj
  Restore completed in 43.01 ms for /Users/bo/src/fable-repl/src/Lib/Fable.Repl.Lib.fsproj.

Parsing ./src/Lib/Fable.Repl.Lib.fsproj...
child proccess errored: Process is terminating due to StackOverflowException.

I'm on a Mac running macOS 10.14.4 and the requested dotnet 2.2.203 of course. On my way I had to change the first line of the build.fsx to use forward slashes (is there a platform independent way to #r btw?) and update the fsevents npm dependency as it didn't build with Node 12.
Unfortunately there's no additional stack for the error, so I don't really know where to investigate further. Just the DirectoryNotFoundException from checking the output dir after.
Any ideas?

Yeah, this is known. Unfortunately it seems a problem with the F# compiler and we cannot fix it on ourn end. Related https://github.com/Microsoft/visualfsharp/issues/6636

Oh alright. Thank you for the info.

Not sure why but this seems to be fixed in latest master after updating dependencies ¯_(ツ)_/¯

It does, thanks!


IntoRaw trait

Every friendly type wrapper should provide means to convert into_raw, maybe through a feature gate (to indicate that this will never be "stable"). For this a trait might do.

This is a thing now. It can be expanded as needed.


remove name clash

@kevgo @allewun
Add a town- prefix to all commands in order to prevent naming conflicts. We can easily alias to keep it as is for normal usage but it can be accessed separately.

I agree with the overall direction. Before we merge this in, let's agree on the details here. I'm not sure running git town-sync is the best user experience. I see these possibilities:

expand the git town command to run our commands. E.g. users run git town sync.

create a gt or town command that calls town-<command>. E.g. calling gt sync or town sync runs git town sync. This could be easily written in Bash. We would be completely out of the Git namespace here, and its even shorter than running git sync.

optionally create git-<command> aliases via a wizard. Users start the wizard via git town aliases. It goes through all Git Town commands, checks if there is an existing git <command> command in the system. If yes it warns the user. If not, it asks the user whether to install the alias.

@charlierudolph @allewun thoughts?

I considered the first two possibilities and found the current approach the fastest and simplest. Minimal change, prevents naming conflicts via a namespace and takes advantage existing git support.

expand the git town command to run our commands. E.g. users run git town sync

The only time I expect someone to type of town-sync is when they have sync aliased to something else. I expect everyone else to use the aliases. Also if we do that then we lose git from autofinding all the subcommands git town adds (like how it suggests sync is a subcommand right now)
This also puts the other config options main-branch etc on the same level. I think we should move those under git town config or something if we go with this approach. I don't see much benefit to this over the current approach though. Yeah we got rid of a single - but at the cost of moving a lot of stuff around and implementing our own redirection.

create a gt or town command that calls town-. E.g. calling gt sync or town sync runs git town sync. This could be easily written in Bash. We would be completely out of the Git namespace here, and its even shorter than running git sync.

I like the natural feeling of git so I didn't think it was worth moving out of the git. We could setup aliases to keep the current approach easy enough but I'm not a bit fan of this. Git already has the aliasing built in so I figured taking advantage of that was the easiest thing we could do.

optionally create git- aliases via a wizard. Users start the wizard via git town aliases. It goes through all Git Town commands, checks if there is an existing git  command in the system. If yes it warns the user. If not, it asks the user whether to install the alias.

Why do we need a wizard? Simple git aliases are really easy. The major case of people just using git-town have it easy and the rest can easily control their own aliases. We can update the current alias script to not overwrite existing aliases.

@kevgo any more thoughts?

Sorry for the delay, I got pulled into many other things. I'm fine with this direction, as long as we make it easy for the user to set up these aliases. Most people don't know what a Git alias is and how to set one up. Ideally some sort of script sets up the aliases. In case of a conflict it proposes a better name for the alias and asks the user what to do.

🚢

I added a script for aliasing but its minimal at the moment and just skips commands where a conflicting alias already exists. We can improve it as desired. Just a heads up that merging this means the next release will be 2.0 since this is a breaking change (although very easy to go back to the previous with the alias script)


do Fiber.yield in deliver_loop

Do Fiber.yield in deliver_loop to allow lavinmq to do other stuff while consuming
https://trello.com/c/DXlvpqpn/273-ui-doesnt-load-when-only-consuming-with-lavinmqperf

Consume rate is way down with this change, guess because the fiber is being yielded after every consumed message?
Especially very low if also publishing at the same time.
With change:
Average publish rate: 187128.1 msgs/s
Average consume rate: 67.8 msgs/s

Without change:
Average publish rate: 65894.9 msgs/s
Average consume rate: 63875.1 msgs/s

When running only one consumer
With change:
Average consume rate: 59335.2 msgs/s

Without change:
Average consume rate: 84173.7 msgs/s

Yes, see this commit: https://github.com/cloudamqp/lavinmq/commit/258e3247eccd924527d4b39340a1eaa7f0375197

Also, you almost always want to benchmark with --release mode.
On Fri, 14 Apr 2023, 15:21 Viktor Erlingsson, @.***>
wrote:

Consume rate is way down with this change, guess because the fiber is
being yielded after every consumed message?
Especially very low if also publishing at the same time.
With change:
Average publish rate: 187128.1 msgs/s
Average consume rate: 67.8 msgs/s
Without change:
Average publish rate: 65894.9 msgs/s
Average consume rate: 63875.1 msgs/s
When running only one consumer
With change:
Average consume rate: 59335.2 msgs/s
Without change:
Average consume rate: 84173.7 msgs/s
—
Reply to this email directly, view it on GitHub
https://github.com/cloudamqp/lavinmq/pull/498#issuecomment-1508496527,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AABL6TS4DKC5NDKVLQJ4ESTXBFFMTANCNFSM6AAAAAAW6M4ZVU
.
You are receiving this because your review was requested.Message ID:
@.***>

Yes, see this commit: 258e324

Okay, nice! 👍
I was playing around with something similar before finishing for the day, but wasn't sure how larger messages would affect something like this.


added userconfig exception for totalcfs<1

This PR closes #117

Can you perhaps add a unit test for this? Otherwise looks good.

sure, added the tests.


Update main.ts

Simplify mapping function

Hi, thank you for the pull request ! I have applied your changes on main.


Images loaded from Disk Cache occasionally display artifacts / corruption.

I'm currently using AsyncDisplayKit + PINRemoteImage libraries in my app.
The PINRemoteImage version is 3.0.0-beta.3 (due to some constraint using in AsyncDisplayKit)
This library is awesome, however, sometimes the cached images look not fully downloaded as the image below:

The function I'm using is
UIImageView *imageView = [[UIImageView alloc] init];
[imageView setPin_updateWithProgress:YES];
[imageView pin_setImageFromURL:[NSURL URLWithString:@"$(my_image_link)"]];

@baby03201 I have a guess as to the cause of that:
Are you running in the simulator. I've seen similar issues when running in the simulator, but never on device.

It is possible that there is an encoding error or other issue with the image file.  Could you send the URL to the image of that size?

On Nov 1, 2016, at 6:13 PM, Garrett Moon<EMAIL_ADDRESS>wrote:
@baby03201 I have a guess as to the cause of that:
Are you running in the simulator. I've seen similar issues when running in the simulator, but never on device.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

@garrettmoon Unfortunately, I couldn't able to regenerate that error in my device now. This screenshot was taken by my friend with his iPhone 6s device. I guess it might happens while the network connection is bad.
@appleguy Thanks for your patience and response!
The images URL link was shown as below:
First image
Second image

I came here looking specifically for information this bug. We just upgraded AsyncDisplayKit from 1.9.81 to 1.9.91 and similarly got PINRemoteImage upgraded to 3.0.0-beta.3. We we started seeing bad images on live devices (not the simulator) and we had not previously been seeing them in the three months we have been using this product. We started focussing on PINRemoteImage when we realize that the problem occurred with images fully managed and rendered by AsyncDisplayKit as well as by code where we called PINRemoteImage directly to load other images.
We are not doing progressive downloads and I have attached one such image that failed. That exact same image worked perfectly on many other devices so it is not a simple data corruption issue.
The AsyncDisplayKit version of this problem using ASNetworkImageNode controlled by a layoutSpecThatFits(). When we call PINRemoteImage directly we are using:
       imageView.pin_setImageFromURL(NSURL(string: look.imageUrl!),                                       completion: { [weak self] (_) in             self?.spinner.removeFromSuperview()         })
We really appreciate all the great work Facebook has done on this amazing product and hope it will continue. We would also appreciate any insight we can get on how to fix this bug.

Hey @SteveGeyer thanks for the info!
Do you happen to know what version of PINRemoteImage you were using previously? 1.9.81 specified beta 2 or better. Can you confirm that's what you were using?
A note: This is actually a Pinterest project and all the full time contributors on AsyncDisplayKit are also Pinterest employees :)

@garrettmoon Yes I was using beta 2.
I apologize for my mistake on credit. PINRemoteImage and AsyncDisplayKit are awesome products. They have made our app truly sing. I really appreciate all the work you and the Pinterest staff has put into them.

@SteveGeyer Awesome, ok, I'll see if there's something that could help attribute the difference. And no worries about the misattribution :)

Here are the changes between beta 2 and beta 3 (https://github.com/pinterest/PINRemoteImage/compare/3.0.0-beta.2...3.0.0-beta.3)
My first guess is it has to do with the cancelation patch. Possibly something getting a callback after cancelation.
https://github.com/pinterest/PINRemoteImage/commit/2cfc232d117a746f2004f61cefc2326149c63443

I am not exactly sure what you may be suggesting. However, the image I provided in my initial post was one that is rendered by AsyncDisplayKit. You are seeing one cell (ASCellNode) made up of three images and some other stuff. This cell's layout is controlled by layoutSpecThatFits() and this cell is in a collection of many cells (ASCollectionView). The AsyncDisplayKit prefetch on the ASCollectionView may in fact be calling cancel on early cells once you scroll too far forward.
One question, if it is being cancelled shouldn't that remove the image completely from the cache? Once we have a damaged image we can't get rid of it. Restarting the app does nothing. Since the app does not have a "clear the cache" feature, the users had to remove the app and reload it to fix the image.
Unfortunately I have not been able to get this to happen on demand. One of my team managed it several time and at least one of our beta testers has seen it. I personally have only seen screenshots. If your theory is correct, is there any way you can suggest that would make provoking it more likely. I would gladly give it a try and get back to you.
My other question is whatever going to beta 5 is likely to fix it. If I revert to 1.9.81 it fetches beta 5 now (sadly 1.9.92 does not -- it is fixed at beta 3). Do you believe beta 5 would help or hurt or just be neutral.
Thanks for all your help.

@SteveGeyer my guess is that beta 5 won't have this issue fixed. I'm going to see if I can set up some conditions which will allow me to reproduce locally.

@SteveGeyer there could be another issue causing this: if an image isn't fully saved to the disk cache when the app is terminated it could result in something like this. It would be good to know if when the issue is reproduced for the first time, if it was pulling the image from the cache or from the network. Not sure an easy way to confirm that either though…

@SteveGeyer actually, if you happen to know if the version of PINCache changed when you upgraded, knowing which version to which would help in investigating.

@garrettmoon I have grabbed my Podfile.lock from the "working" system and edited out unnecessary pods. Below are what I believe are the relevant lines. It shows PINCache 2.2.2.
PODS:

AsyncDisplayKit (1.9.81):

AsyncDisplayKit/PINRemoteImage (= 1.9.81)

AsyncDisplayKit/ASDealloc2MainObject (1.9.81)
AsyncDisplayKit/Core (1.9.81):

AsyncDisplayKit/ASDealloc2MainObject

AsyncDisplayKit/PINRemoteImage (1.9.81):

AsyncDisplayKit/Core
PINRemoteImage/iOS (>= 3.0.0-beta.2)

PINCache (2.2.2)
PINRemoteImage/Core (3.0.0-beta.2):

PINCache (>= 2.1)

PINRemoteImage/iOS (3.0.0-beta.2):

PINRemoteImage/Core

pop (1.0.9)

I talked to my coworker who is very experienced with the app and he assures me that it did not crash nor did he stop the app part the way through. I completely trust his report. He also kept his device in its broken state so I could bring up a debugger and run code against the cache. Would this help you?
As to your question on where the image was coming from. It was most likely coming from the network and not the cache. None of the images preexist in the app. The image in my example above would have been from AsyncDisplayKits's prefetch of cells and that would have initially come from the network. You see the TOMORROW at the top. I talked to him and he had just flipped to the tomorrow page to do QA on tomorrow's content delivery. He would have flipped to this page, waited for the screen to load from the network, and then reviewed this piece of content. It got stuck downloading the image.

@SteveGeyer very valuable information, thank you! I'm investigating this as we speak :)

@stevegeyer there would be one more valuable piece of information you could gather for me. Specifically the file that's stored in the cache. You should be able to get to it by calling objectForKey with the offending URL on the cache object of [PINRemoteImageManager sharedImageManager] (the key you pass in can be retrieved by calling [[PINRemoteImageManager sharedImageManager] cacheKeyForURL:URL processorKey:nil]
Knowing the URL would also be helpful.
Finally, one more question: are you attempting to use PINRemoteImage anywhere else in the app directly? Or are you just using the ASDK implementation?

I am off to a meeting right now, but will grab this data and answer your questions in a couple of hours.

@garrettmoon I will attempt to get the local cache file tomorrow when I
have access to the actual device that failed. I will also try and get you
the URLs involved.
We are definitely calling PINRemoteImage directly elsewhere in the app. In
fact those calls are likely happening at the same time as ASDK. The URLs
used by ASDK and direct calls will different since ASDK is handling the
thumbnails while the direct calls are fetching the full image for later use.
On Thu, Nov 3, 2016 at 6:26 PM, Garrett Moon<EMAIL_ADDRESS>wrote:

@stevegeyer https://github.com/stevegeyer there would be one more
valuable piece of information you could gather for me. Specifically the
file that's stored in the cache. You should be able to get to it by calling
objectForKey with the offending URL on the cache object of [PINRemoteImageManager
sharedImageManager] (the key you pass in can be retrieved by calling [[PINRemoteImageManager
sharedImageManager] cacheKeyForURL:URL processorKey:nil]
Knowing the URL would also be helpful.
Finally, one more question: are you attempting to use PINRemoteImage
anywhere else in the app directly? Or are you just using the ASDK
implementation?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pinterest/PINRemoteImage/issues/260#issuecomment-258292858,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAxKS5HFEzaAI7vgGrVlobiupcQZoS-Eks5q6l-egaJpZM4KejcV
.

@garrettmoon The damaged image that I showed in my original post has "repaired" itself since yesterday. I assume that its expiration timed out and it was refetched. However, there is another image that is failing and I have attached an image of this failure to the bottom of this message.
Here is the source URL for this failing image: https://s3.amazonaws.com/lookbk/imgs/01/fe/dD/tP/01fedDtP-LRfQaBOI6.jpg
I ran [[PINRemoteImageManager sharedImageManager] cacheKeyForURL:URL processorKey:nil] in the app and I get the same URL back (https://s3.amazonaws.com/lookbk/imgs/01/fe/dD/tP/01fedDtP-LRfQaBOI6.jpg).
At the same time as we were fetching these thumbnails from ASDK, we are likely fetching larger versions of these images with a direct calls to PINRemoteImage. In this example, the image can be found at https://s3.amazonaws.com/lookbk/imgs/01/fe/dD/tP/01fedDtP-dofJvaQiV.jpg
While I was in this code I realized that I do limit the size of the cache:

let imageCache = PINRemoteImageManager.sharedImageManager().cache.diskCache
imageCache.byteLimit = imageCacheSize
imageCache.ageLimit = maxCacheTime`

The cache byteLimit was 495MB and the ageLimit is 2 weeks. The specific device that failed had been used for a long time so it is it possible we are hitting the size limit while loading the image?  Could this be an issue?
I hope all this helps. Thanks again for your help.

Can you also attach the corrupted image from the cache? so if cacheKeyForURL returned just the URL, you should be able to get the NSData out by calling [[[PINRemoteImageManager sharedImageManager] cache] objectForKey:@"https://s3.amazonaws.com/lookbk/imgs/01/fe/dD/tP/01fedDtP-LRfQaBOI6.jpg"]
Not sure the easiest way to get the NSData off the phone… Maybe try and write it to the Documents directory and get at it with iTunes file sharing (http://stackoverflow.com/questions/6029916/how-to-enable-file-sharing-for-my-app)? That's a bit of a pain, but I'd really appreciate it, I'm hoping it can confirm my theory.

@garrettmoon Sadly we reset the broken device. We will try and recreate the error and capture that image (I am all set up to do the capture once we have another failure).

@SteveGeyer Just wanted to say thanks for how proactive you've been on bringing information to this thread.  It's extremely helpful.  We haven't seen a corruption issue like this in Pinterest, so it is really important that we capture any clues we can to prevent it for others.
Could you share a bit more about how you're using PINRemoteImage?  I have a feeling that the issue is caused by interacting with two different PINRemoteImageManagers at the same time, about the same resource — e.g. one might be trying to read it from the cache, while the other is writing.  This may not be the case (or a problem), since I think they share a disk cache instance, but it could be the difference in configuration that means Pinterest doesn't see the issue.  If it is related to having multiple instances, we will have a much better lead on fixing it.
@garrettmoon would there be any benefit in using the ASPINRemoteImageManager +sharedInstance?  IIRC that's how you can get the instance that ASDK uses.  Maybe that would avert the issue, and if so, we'd have a good lead.

I am going to overshare here in the hopes that what I write sparks ideas on your side. I have grepped our source code and I will outline each direct call we make to PINCache and PINRemoteCache. Our app is written totally in Swift. I doubt that is significant, but maybe your team knows something I don't. We are also big fans of ASDK and PINCache/PINRemoteCache and we are using all three products directly (and of course ASDK uses PINCache/PINRemoteCache internally).
For this discussion it is important to know that our app shows a list of potential articles you can view using ASDK. ASDK, through ASNetworkImageNode()s with their URL set,  is responsible for fetching thumbnail images used in the summary pages. The intelligent preloading logic of ASKD is used by us to fetch the full images of the article using PINRemoteImage. The important point here is that ASDK fetches thumbnails while our manual calls to PINRemoteImage fetch full sized images. They are fundamentally different URLs. There is a single exception to this rule that I will get to later.
Since we are using PIN caches for both JSON and images, we set up cache limits at startup:

let jsonCache = PINCache.sharedCache().diskCache
jsonCache.byteLimit = jsonCacheSize
jsonCache.ageLimit = maxCacheTime
let imageCache = PINRemoteImageManager.sharedImageManager().cache.diskCache
imageCache.byteLimit = imageCacheSize
imageCache.ageLimit = maxCacheTime

jsonCacheSize is 5MB, imageCacheSize is 495MB and ageLimit is 2 weeks.
The two lines of code we use store and retrieve JSON are:

PINCache.sharedCache().objectForKey(key)
PINCache.sharedCache().setObject(data, forKey: key)

Intelligent preloading has us prefetching the full size images the the following line. We usually fetch between 10-20 images each time:

uuids = PINRemoteImageManager.sharedImageManager().prefetchImagesWithURLs(imageURLs)

If the preloading logic says we are abandoning this article, we cancel any remaining fetches with:

for uuid in uuids {
PINRemoteImageManager.sharedImageManager().cancelTaskWithUUID(uuid)
}

Note again that these are full sized image with different URLs from the thumbnails fully managed by ASDK.
Like I said earlier, we have one UI controller that prefetches the same thumbnails that ASDK fetches. It uses:

uuids = PINRemoteImageManager.sharedImageManager().prefetchImagesWithURLs(imageURLs)

Here the imageURLs list is thumbnails. However, these thumbnails are not being used in active articles. So it is not possible that they duplicate one of the thumbnails being fetched by ASDK.
For completeness, If that controller sees the cell leaving the view it cancels the fetch with.

for uuid in uuids {
PINRemoteImageManager.sharedImageManager().cancelTaskWithUUID(uuid)
}

This represents all calls that we make directly to PINCache and PINRemoteImage. While I believe we are frequently mixing calls of ASDK fetches with direct fetches, I believe the URLs used by ASDK are different than the ones we manually fetch using PINRemoteImage.
I will gladly attempt to answer any other questions you may have.

This is very helpful Steve, thank you!
Before diving in with further suggestions, I'll let Garrett think through this feedback - and especially whether multiple ImageManagers could interact in a negative way (even with distinct URLs).
One tip: you can achieve the same preloading effect by using an offscreen ASNetworkImageNode that you trigger to perform its preloading.  You can do this by setting its .interfaceState to Preload.  We could add an API to make that a bit easier :).
There's also ASMultiplexImageNode, which works especially well if you want to take an image / thumbnail and zoom it up in an animation; it can perform the downloading and decoding of a higher resolution version asynchronously, even as the onscreen version is being animated.  It also uses the best-fit disk cache version while trying to load the high res, so it would automatically show an upscaled thumbnail version if the high-res request is still in flight (this might be desirable for your app).

On Nov 4, 2016, at 6:40 PM, SteveGeyer<EMAIL_ADDRESS>wrote:
I am going to overshare here in the hopes that what I write sparks ideas on your side. I have grepped our source code and I will outline each direct call we make to PINCache and PINRemoteCache. Our app is written totally in Swift. I doubt that is significant, but maybe your team knows something I don't. We are also big fans of ASDK and PINCache/PINRemoteCache and we are using all three products directly (and of course ASDK uses PINCache/PINRemoteCache internally).
For this discussion it is important to know that our app shows a list of potential articles you can view using ASDK. ASDK, through ASNetworkImageNode()s with their URL set, is responsible for fetching thumbnail images used in the summary pages. The intelligent preloading logic of ASKD is used by us to fetch the full images of the article using PINRemoteImage. The important point here is that ASDK fetches thumbnails while our manual calls to PINRemoteImage fetch full sized images. They are fundamentally different URLs. There is a single exception to this rule that I will get to later.
Since we are using PIN caches for both JSON and images, we set up cache limits at startup:
let jsonCache = PINCache.sharedCache().diskCache
jsonCache.byteLimit = jsonCacheSize
jsonCache.ageLimit = maxCacheTime
let imageCache = PINRemoteImageManager.sharedImageManager().cache.diskCache
imageCache.byteLimit = imageCacheSize
imageCache.ageLimit = maxCacheTime
jsonCacheSize is 5MB, imageCacheSize is 495MB and ageLimit is 2 weeks.
The two lines of code we use store and retrieve JSON are:
PINCache.sharedCache().objectForKey(key)
PINCache.sharedCache().setObject(data, forKey: key)
Intelligent preloading has us prefetching the full size images the the following line. We usually fetch between 10-20 images each time:
uuids = PINRemoteImageManager.sharedImageManager().prefetchImagesWithURLs(imageURLs)
If the preloading logic says we are abandoning this article, we cancel any remaining fetches with:
for uuid in uuids {
PINRemoteImageManager.sharedImageManager().cancelTaskWithUUID(uuid)
}
Note again that these are full sized image with different URLs from the thumbnails fully managed by ASDK.
Like I said earlier, we have one UI controller that prefetches the same thumbnails that ASDK fetches. It uses:
uuids = PINRemoteImageManager.sharedImageManager().prefetchImagesWithURLs(imageURLs)
Here the imageURLs list is thumbnails. However, these thumbnails are not being used in active articles. So it is not possible that they duplicate one of the thumbnails being fetched by ASDK.
For completeness, If that controller sees the cell leaving the view it cancels the fetch with.
for uuid in uuids {
PINRemoteImageManager.sharedImageManager().cancelTaskWithUUID(uuid)
}
This represents all calls that we make directly to PINCache and PINRemoteImage. While I believe we are frequently mixing calls of ASDK fetches with direct fetches, I believe the URLs used by ASDK are different than the ones we manually fetch using PINRemoteImage.
I will gladly attempt to answer any other questions you may have.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/pinterest/PINRemoteImage/issues/260#issuecomment-258583089, or mute the thread https://github.com/notifications/unsubscribe-auth/AAigA_Y0F02LPkjIrEOaChk1Ql71YKfdks5q696QgaJpZM4KejcV.

@garrettmoon I am set up to dump a broken image as you previously requested. Unfortunately our failed images were automatically refreshed after their time-to-live expired and we have not seen this problem reoccur. If we see another failure I will capture the image for you.
This brings up some potential evidence about this bug. We saw three failures across two long term users shortly after they upgraded to our newest version with the upgraded PINRemoteImage. We have not had any other errors reported (although our beta users are not always the best at reporting issues).
So I wondered if the issue could be related to upgrading the library. Did the cache storage format subtly change in a manner that has a small hiccup with pre-existing and fully loaded caches? My other ponder was if it could be related to the cache being at its max limits. Both of these users would have had the image cache at their limit with each new image requiring the cache eject one or more previous images. Maybe there is a subtle issue there. Finally, since these failures only happened in one short window of time, I wonder if there was some subtle issue with our hosted images and them causing the caching code to receive some unusual network error. All of these are just idle speculations from an interested user, so use or ignore these as appropriate.
We are getting near our initial release and have to decide what to do next. Do we roll back our versions of ASDK and the PIN caches? Do we move forward on the assumption these were some special kind of failure that won't happen regularly to normal users (either it was upgrade related or network failure related)? Or do we move forward assuming you will find and post a fix shortly? Any feedback you can bring to this decision would be greatly appreciated.
I thank you again for all the work you are doing to resolve this.

@SteveGeyer All reasonable theories and to be honest I still haven't been able to find any conclusive to point to anything. As @appleguy mentioned, we haven't seen reports of these errors at Pinterest, but we're on latest master (or very close) of ASDK, PINRemoteImage and PINCache.
To be frank, I wouldn't rely on us being able to post a fix shortly. I believe this is a very important issue to get to the bottom of, but without a solid way to reproduce, I'm not sure I can promise a quick fix.
If I were in your situation I'd probably monitor to see if it comes up again. If it doesn't stick with what you have, if it does I would attempt to mitigate what is possibly the cause:

Use only one instance of PINRemoteImage, the one created by ASPINRemoteImageDownloader. We do this in Pinterest by creating a category which we use instead of PINRemoteImageManager:

+ (PINRemoteImageManager *)sharedImageManager
{
    return [[ASPINRemoteImageDownloader sharedDownloader] sharedPINRemoteImageManager];
}

And please keep reporting back :)

@garrettmoon We will consider your advise as we move forward. If anything else happens at our end I will post what happened and what we did. Thanks.

@SteveGeyer thank you! All the info you've given us is going to definitely narrow down solving this issue. Hopefully very soon :)

I am experiencing the same issue in a country where internet connection is slow, unreliable and when there are multiple concurrent image requests (around 60 images per screen view).

@boreys what version of PINRemoteImage are you using? Do you have a test app you can reliably reproduce the issue on? If so, I should be able to simulate a similar internet connection.

I'm using PINRemoteImage 3.0.0-beta.3 and 'AsyncDisplayKit', '~>1.9.90' you can download my app from the store and follow this step, it most likely produce the result ( a couple broken images):
1- Download the app: https://itunes.apple.com/us/app/khmer.be/id1138606309?ls=1&mt=8
2- Create an account and follow some users to generate around 60 images or more on the "News Feed" page.
3- Log out, uninstall the app and then reinstall the app so that the cache is removed from device.
5- Turn Network Conditioner on with the worst condition: slow and interrupting
6- Log in to the app to download fresh images on the News Feed, which should aggressively download the images and generate some broken ones
Mean while I'm gonna try to debug this library myself too.

@garrettmoon  I am also facing the same issue. The images are getting corrupted in the same fasion as the screenshot attached by @boreys

Whoo! I think we finally got this one licked! I'll update this thread when I've pushed a new beta for you all to try.

Going to close this as it should be fixed in PINRemoteImage (3.0.0-beta.7) & ASDK (2.0-rc.2). Please give it a try and if you hit this again mention it here.

Thanks for fixing the bug. I will be trying the fix out shortly.

Just wanted to let you know we experience this from time to time. I just got it on this file:
https://170213-images.liketoknow.it/791cd8d0-19a4-11e7-9fed-0242ac110002?fm=pjpg&q=75&w=345&fit=max
I mounted the phone with iExplorer and pulled it off disk:

This screenshot represents how it renders on the phone:

We are on:
PINRemoteImage (3.0.0-beta.8)
It is pretty rare.

This is a very helpful report - thanks for sending it. I know that beta.9
is now available, but am not sure if it has changes intended to resolve
this. It would be worth updating just in case, but I think the frequency of
this issue has reduced even though it is still sometimes happening.
I'm pretty sure that if the app crashes or is killed (as might happen when
developing / debugging too), we could write a corrupted file if it is in
progress at the time of the kill. It's an interesting question on how we
might detect and discard these.
On Thu, Apr 27, 2017 at 6:42 PM emablekos<EMAIL_ADDRESS>wrote:

Just wanted to let you know we experience this from time to time. I just
got it on this file:
https://170213-images.liketoknow.it/791cd8d0-19a4-11e7-9fed-0242ac110002?fm=pjpg&q=75&w=345&fit=max
I mounted the phone with iExplorer and pulled it off disk:
[image: busted]
https://cloud.githubusercontent.com/assets/299982/25510985/08f0312a-2b92-11e7-8e22-ca479843f03f.jpg
This screenshot represents how it renders on the phone:
[image: image]
https://cloud.githubusercontent.com/assets/299982/25510996/260da6ac-2b92-11e7-912f-a8630b76c682.png
We are on:
PINRemoteImage (3.0.0-beta.8)
It is pretty rare.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pinterest/PINRemoteImage/issues/260#issuecomment-297884669,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAigAw660d_tD1eLBYti_s_FDI8B8Sclks5r0UP7gaJpZM4KejcV
.


Fix external forks for CDK nag

Feature or Bugfix

Bugfix

Detail
See issue #766
Relates
#766
Security
Please answer the questions below briefly where applicable, or write N/A. Based on
OWASP 10.

Does this PR introduce or modify any input fields or queries - this includes
fetching data from storage outside the application (e.g. a database, an S3 bucket)?

Is the input sanitized?
What precautions are you taking before deserializing the data you consume?
Is injection prevented by parametrizing queries?
Have you ensured no eval or similar functions are used?

Does this PR introduce any functionality or component that requires authorization?

How have you ensured it respects the existing AuthN/AuthZ mechanisms?
Are you logging failed auth attempts?

Are you using or adding any cryptographic features?

Do you use a standard proven implementations?
Are the used keys controlled by the customer? Where are they stored?

Are you introducing any new policies/roles/users?

Have you used the least-privilege principle? How?

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

The test works but it still requires some testing in a real AWS deployment.


Regression in TOML LocalDate Format Support

What version of Hugo are you using (hugo version)?

Regression introduced in hugo v0.87.0-B0C541E4 linux/amd64 BuildDate=2021-08-03T10:57:28Z VendorInfo=gohugoio
Regression exists in hugo v0.99.1-d524067382e60ce2a2248c3133a1b3af206b6ef1 linux/amd64 BuildDate=2022-05-18T11:18:14Z VendorInfo=gohugoio
Last confirmed release where this pattern works is hugo v0.86.0-41C6C52E linux/amd64 BuildDate=2021-07-21T09:53:14Z VendorInfo=gohugoio

Issue
This is quite a minor issue as there is a solution. Feel free to close this if I've misunderstood and this isn't a valid regression.
In Hugo before v0.87 you could use the .Format method on a toml.LocalDate date. Since then with the new TOML backend and updated .Format functionality it errors.
For hugo <= 0.86, given the toml:
date = 2021-10-19

This template renders correctly:
{{ with index . "date" }}

Date

    <time
        datetime='{{ .Format "2006-01-02" }}'>
      {{ .Format "2006-01-02" }}
    </time>

{{end}}

From hugo >= 0.87 it errors with:
Error: Error building site: failed to render pages: render of "page" failed: "/path/to/hugo/template.html:42:25": execute of template failed: template: hugo/template.html:42:25: executing "main" at <.Format>: can't evaluate field Format in type toml.LocalDate

However, calling time.Format directly and passing in toml.LocalDate as an argument does correctly render.
{{ with index . "date" }}

Date

    <time
        datetime='{{ time.Format "2006-01-02" . }}'>
      {{ time.Format "2006-01-02" . }}
    </time>

{{end}}

Shouldn't toml.LocalDate (which can be coerced into a time.Time) be treated as a just like regular a TOML Offset Date-Time (which I think are automatically time.Time in go-toml).

I don't think we can/want to fix this, though.

When using toml.LocalDate and toml.LocalDateTime values in the four predefined front matter date fields, the values are converted to time.Time, so the .Format method works as expected.
This issue is limited to custom front matter date fields, and date values from local and remote data files.
As noted here, it would be nice if pelletier/go-toml had an option to unmarshal toml.LocalDate and toml.LocalDateTime to time.Time in UTC. Or we walk the tree and do it ourselves.
We've had to make adjustments here and there to accommodate toml.LocalDate and toml.LocalDateTime values:

https://github.com/gohugoio/hugo/pull/9981
https://github.com/gohugoio/hugo/pull/11706

It would be nice if we could just handle it while unmarshaling.


Disable SSL certificate validation for PHP > 5.6.0

The SSL certificate validation is already disabled when using PHP<5.3.7 as it's configured in cURL options (Http.php:161).
However when PHP>5.3.7 is used, native fopen is used as the http client, and it's assumed that SSL is disabled by default (line 112).
However as of PHP version 5.6.0, the SSL certificate validation is enabled by default (reference: http://php.net/manual/en/context.ssl.php), and this cause user with PHP >= 5.6.0 not to be able to use the sdk library.
This pull request will fix the issue for these users, by explicitly disable certificate validation in http context.
I have already signed the individual contributor agreement. Thanks in advance for considering the change.

I made that pull request over 2 weeks before... https://github.com/splunk/splunk-sdk-php/pull/40

@adamziaja
First, I haven't check other pull request when submitting mine.
Second, this pull request is far is far better than yours; 1. Better description of problem 2. Reference to related documentations 3. Same styling as the project 4. Comments
Third, #36 is opened 3 month before yours, what is the point?

+1 to please pull this! - am using the sdk via the https://github.com/easylo/splunk-sdk composer package and need this

@code-guru @anthonysomerset @adamziaja How soon do you all need a release of the SDK with this change?

@shakeelmohamed we already applied the patch to our code, and have it in our internal repo, however we wanted the change in official sdk, so if want to update splunk-sdk in future, we don't lose the fix.
Thank you for merging the change.


checking hdf file always fail?

Getting this messages for all file.  is this check really working?
/home/yosuke/input_data/raster/e4ftl01.cr.usgs.gov/MOLT/MOD44B.061/2022.03.06/MOD44B.A2022065.h35v10.061.2023122194549.hdf ... corrupted, downloading again
https://e4ftl01.cr.usgs.gov/MOLT/MOD44B.061/2022.03.06/MOD44B.A2022065.h35v10.061.2023122194549.hdf

they are using md5sum now

#62


Model Performance Degradation When Training Llama-2-7b-chat-hf with DPO on Anthropic Dataset

Hello,
I am currently facing a challenge while training a fine-tuned NousResearch/Llama-2-7b-chat-hf model using the (DPO) method.  The training dataset I'm utilizing is the https://huggingface.co/datasets/Anthropic/hh-rlhf, and the script I'm using is from the Hugging Face TRL repository examples: dpo_llama2.py.
Training Arguments:
{
    "beta": 0.5, // Also tried with 0.1
    "learning_rate": 0.0005,
    "lr_scheduler_type": "cosine",
    "warmup_steps": 100,
    "weight_decay": 0.05,
    "optimizer_type": "paged_adamw_32bit",
    "per_device_train_batch_size": 1,
    "per_device_eval_batch_size": 1,
    "gradient_accumulation_steps": 16,
    "gradient_checkpointing": true,
    "lora_alpha": 32,
    "lora_dropout": 0.05,
    "lora_r": 16,
    "max_prompt_length": 512,
    "max_length": 1536,
    "num_train_epochs": 2,
    "logging_steps": 10,
    "ignore_bias_buffers": false
}

Issue:
Before the DPO training,  the fine-tuned model is out-performing the base NousResearch/Llama-2-7b-chat-hf on the alpaca_eval dataset.
After training, contrary to expectations, the model's performance deteriorates instead of improving. This is perplexing as I anticipated an enhancement in performance post-training.
Questions and Assistance Needed:

What could be causing the degradation in model performance?
Are there any specific parameters or configurations in the training setup that might be contributing to this issue?
Is there any additional information or diagnostic data that would be helpful to share for troubleshooting this problem?

Any insights or suggestions would be greatly appreciated.

From what I heard is the HH dataset not super high quality, but I defer to @lewtun and @edbeeching who did a lot of experiments with the DPO and different datasets.

I also met the situation when using dpo based llama2-13b-SFT.

why endless repetitions？


Support initializer before LifeCycleManager.start

There would be a situation that some initializers' operation must be finished before @PostConstruct( LifeCycleManager.start) methods are called. For example, some plugins must be loaded before other instances start services.

In Platform, I created an @AcceptRequests lifecycle annotation which is guaranteed to be invoked after all @PostConstruct are complete. I then moved HttpServer.start() to the new annotation.
The idea is that one specifically marks the methods that cause the server to start taking external work.

I've always disliked that we conflate "post-construct" (which is part of object creation and meant to work around java memory model constraints) and "service start". I like the idea of a different annotation for this purpose.

proofpoint/platform@41cafe0f0041238dc9158d0660359c5950a1d3be

Brilliant, I also like the idea of a new annotation.


Hide the token patterns we look for in table!

The generated documentation for the table! macro is really bad. Macros
document all of the patterns that they match on. For complex macros such
as this one, this is not useful information to the user.
This was maybe useful back in 0.1 when the docs looked like
https://docs.rs/diesel/0.1.0/diesel/macro.table!.html, but now it's just
a bunch of opaque macro soup that the user doesn't care about. The
actual documentation is buried more than two full page scrolls down.
This makes the documented token pattern as small as possible, so that
the actual documentation can be front and center.
Before:

After:

Immediately found code which validates my concerns in crates.io. Approximately this:
use schema::versions;

#[derive(Identifiable)] // <--- This requires `schema::versions` be in scope for the module
pub struct Version {
    // ...
}

impl Version {
    //       v---- This variable name will always error if `schema::versions` is in scope for the module
    fn max(versions: Vec<Self>) -> Self {
        // This function does not do anything Diesel related
    }
}

If we made this change, I'd probably end up fixing our test suite by either limiting the imports we do from schema or always importing them at the function level instead of the module level. The problem is, we have code that enforces that you do the import at the module level. With this change, you could no longer use a variable with the same name as the table anywhere in the file, even though it's often the most convenient variable name for a collection in functions which have nothing to do with Diesel.

Yeah so I think this is pretty much dead unless we're ok with this change as is. I explored if we could have all our derives which take the table name take a path instead, and it's basically impossible unless we change Rust to allow a path fragment to be followed by ::


Defination of positive and negetive samples in foveabox

As illustrated in the foveabox paper, 2 hypreparameters sigma1 and sigma2 are used to define positive and negetive samples. However, i can only find one sigma in config files of foveabox. So where is another sigma? Did i missed something here?

Code released by the author is a little different from the paper.


"Result" project is passing now

https://github.com/apple/swift/issues/70970

@swift-ci please test


Convert Pasted Line Endings and then send to compty

Summary of the Pull Request
This is from a PR request by @d-bingham. However, there have been 89 commits since then, and there are merge conflicts that have not been attended to in a month, and I think converting pasted line endings is a neat idea, so this PR has the changes in PR#1094, but rebased to be current.

References
PR#1094

PR Checklist

[x] Closes #1091
[x] CLA signed. If not, go over here and sign the CLA
[x] Tests added/passed
[x] Requires documentation to be updated
[x] I've discussed this with core contributors already. If not checked, I'm ready to accept this work might be rejected in favor of a different grand plan.

Detailed Description of the Pull Request / Additional comments
@d-bingham said

The new option, convertPasteLineEndings does what it says -- replacing Windows-space CRLF pairs with Unix-space LFs in text pasted to the console. This option applies to both the keyboard-shortcut paste and the right-click paste. Without this most multiline text pasted into Terminal will be "double-spaced" due to the Windows-style CRLF pairs. Furthermore, any multiline text copied from Terminal (in trim whitespace mode) generates CRLF pairs, which guarantees double-spacing when copying from a WSL Terminal session.
Changes:

Added TermControl::_SendPastedTextToConnection which adds a pre-processing layer on top of _SendInputToConnection to allow line-ending conversion

Various code in Profile/TerminalSettings to support the new option "convertPasteLineEndings" (optional, defaults to false)

Validation Steps Performed
I performed manual and automated testing to ensure this would not break any current code.

cc: @zadjii-msft, I would like to get your input, as conpty does no such thing.

@DHowett May I please get your opinion on this too?

 It looks like they all (unconfigurably!) transform \n to \r (on Linux) and \r\n to just \r (on Windows). I think we can safely do that and not need to introduce a new setting.
Here's why I think that: Windows Terminal wants to be a standards-compliant--or, as compliant as it can be--terminal emulator. It should act in every way like every other terminal emulator. If that means that it only ever sends an enter keypress as CR (\r), so be it.
By making Windows Terminal compliant with specified standard (search for | CR to get to the keyboard section), we can leave the compatibility work of translating CR back into CRLF for Windows applications in ConPTY.
Also, unrelatedly, it is very hard for us to accept pull requests that also make unrelated code style and formatting changes. Please try to refrain from doing that!
Thanks.

So we should put this functionality in Conpty? got it.

Wait no! Conpty handles this correctly (translating \r to Enter), so Terminal should always translate \r\n to \r. We don’t need a configuration option for it :)

Ok, got it. so no need to do this here, right?

Right, we just need to make convertPasteLineRndibgs the default and remove the option to disable it.

I did mean “endings” 😄

oh wow, we need this! 😱

So should I go and do those changes? Or is there work being done already

@pi1024e we’d love that! No work is ongoing here.

I'm going to close this one out -- please let me know if you still want to work on the final iteration! If not, I might make an attempt at it myself. :smile:

I am going to open a new PR following your advice. Thank you!


Legal text component

Preview:

design q: how different is the ! in the icon to the one in tabular or normal transport? if not much, we could just use that instead of .png or .svg

Would this be better named: legal-consequence?

The original version:

Using transport for the ! and the large step icon circle (38px x 38px):

Nice!

How's this look cross browser?
Does a assistive tech read the bang out?
What happens when you zoom?

As a side note, what is the scope of components that should be on GOV.UK Frontend?
Is there a criteria that means it will be useful and reused across Government, my gut with this component would be that it seems pretty specific to GOV.UK Publishing?

Hi Nick,
I'll need to take screenshots for the first and add aria-hidden="true" for the second.
GOV.UK Frontend's initial scope is to include components for all the examples we provide as part of GOV.UK elements, here's the legal text example:
https://govuk-elements.herokuapp.com/typography/#typography-legal-text
This is used by services, often for declaration pages - for example, I confirm that the information I have given is true... and this component will be used underneath to reinforce that the declaration above has a legal consequence.

Voiceover reads "Warning You can be fined up to £5,000 if you do not register".

It looks like it should, apart from in IE8 which doesn't support border-radius so we get a big black square rather than a circle.
IE8:

IE11:

Let's go with this for now. If we decide when we start the work on icons, that we would like to use an inline SVG for this icon to be consistent with icons used elsewhere - then we can change it at that point.


Recognize 1-time passcodes, address, Phone numbers  etc.

Is it possible to Recognize 1-time passcodes and add them to the clipboard?
Also is possible to recognize emails address or phones numbers in a messages and make them hyperlink?
Can there be an option added to be able to copy part of message instead of all of it?

You can press and hold the copy button to bring up a dialog that you can copy a selection from. We might want to add an explicit option in the more options menu

What platform of the client are you using?

What platform of the client are you using?

I’m using the android version.

What platform of the client are you using?

I’m using the android version.

You can enable the phone number and email linking in settings > conversation settings > smart suggestions


remove amd64 keyword

https://github.com/open-cluster-management/backlog/issues/15090#issuecomment-901551267
Signed-off-by: Song Song Li<EMAIL_ADDRESS>

/lgtm


Library is missing dedicatedPod support

/kind enhancement
/area library
Summary
dedicatedPod is defined in devfile spec, for creating a separated pod for a single container component.
devfile library currently does not support dedicatedPod. library need to support that if there is an actual use case turns that field on.
generator for service/deployment should support dedicatedPod and returns a list of services & deployments

closing as rotten (created > 2 years). Feel free to re-open if it's not a requirement.


Unable to deploy to function app VS Code

Does this occur consistently? Yes
Repro steps:
Deploy to Function app...
Action: azureFunctions.deploy
Error type: 409
Error Message: There is a deployment currently in progress. Please try again when it completes.
Version: 0.22.0
OS: win32
OS Release: 10.0.18362
Product: Visual Studio Code
Product Version: 1.44.2
Language: en

Call Stack
extension.bundle.js:309:399876extension.bundle.js:309:399876
retryCallback extension.bundle.js:225:114550
retryCallback extension.bundle.js:157:31063
extension.bundle.js:157:32676extension.bundle.js:157:32676
handleRedirect extension.bundle.js:171:7752
extension.bundle.js:157:29848extension.bundle.js:157:29848
Request._callback extension.bundle.js:1:669090
Request.n._callback.n.callback.n.callback extension.bundle.js:201:4483
Request.emit events.js:203:13
Request.<anonymous> extension.bundle.js:201:17592
Request.emit events.js:208:15
IncomingMessage.<anonymous> extension.bundle.js:201:16382
Object.onceWrapper events.js:291:20
IncomingMessage.emit events.js:208:15
endReadableNT _stream_readable.js:1168:12
processTicksAndRejections task_queues.js:77:11

Hi @brgsstm
Thanks for reporting.  This error usually comes up if you try to deploy to the same app back-to-back.  Have you tried waiting and deploying later?  Restarting the app can help with this issue as well.
If that doesn't work, could you try creating a new function app and deploying to that one?

waited and deployed successfully - frustrating.  you can close


Handle symbol ID links

Following up on https://github.com/TypeStrong/typedoc/issues/2700#issuecomment-2356587649 please add support for symbol ID links.

Confirmed. The fix worked for me.


Need a LICENSE file

This repository needs a LICENSE file to clarify conditions for re-use.
My own suggestion is always an MIT or Apache 2 license, but the use of IDSA components may require a GPL license. In any case, you should let your downstream users know what's what. Having this in a LICENSE file also helps github and other automated tools understand and check your constraints.
Note that some licenses of your dependencies may also require a NOTICE file.

License information added to project

And Github recognized the license correctly!  Yay.
Nice.


Unable to build Web example

I tried to clone and build webJsApp to see library in action since I'm exploring it to use it.
I encountered a couple for issues

Incorrect Import on line no 5 here
After resolving #1 issue I got below error for copy gradle task on Android Studio HeadgHog Canary 3

`bash
qrose % ./gradlew example:webJsApp:jsBrowserDevelopmentRun
Task :example:webJsApp:jsDevelopmentExecutableCompileSync FAILED
FAILURE: Build failed with an exception.

What went wrong:
A problem was found with the configuration of task ':example:webJsApp:jsDevelopmentExecutableCompileSync' (type 'DefaultIncrementalSyncTask').

Gradle detected a problem with the following location: '/Users/mohammedhamid/Documents/Projects/KMP/qrose/example/webJsApp/build/processedResources/js/main'.
Reason: Task ':example:webJsApp:jsDevelopmentExecutableCompileSync' uses this output of task ':example:webJsApp:copyJsResourcesWorkaround' without declaring an explicit or implicit dependency. This can lead to incorrect results being produced, depending on what order the tasks are executed.
Possible solutions:

Declare task ':example:webJsApp:copyJsResourcesWorkaround' as an input of ':example:webJsApp:jsDevelopmentExecutableCompileSync'.
Declare an explicit dependency on ':example:webJsApp:copyJsResourcesWorkaround' from ':example:webJsApp:jsDevelopmentExecutableCompileSync' using Task#dependsOn.
Declare an explicit dependency on ':example:webJsApp:copyJsResourcesWorkaround' from ':example:webJsApp:jsDevelopmentExecutableCompileSync' using Task#mustRunAfter.

Please refer to https://docs.gradle.org/8.0/userguide/validation_problems.html#implicit_dependency for more details about this problem.
`

Thanks. Looks like broken resources configuration in web example. I'll fix it. It will work normally if you add it to your project as a gradle dependency

I can raise this PR, library looks good.

Sure thing, if you want


Add endpoint to calculate Zero Coupon Bond Effective Yield

Add an endpoint to Zero Coupon Bond Effective Yield.
Create a function in ./helpers/functions.py file for the calculation.
Map it with an endpoint GET /zero_coupon_bond_effective_yield in the main.py file with properly.

Go ahead @vamsikrishnarh7 :))


Serialize cleaning up znodes for old db versions and avoid a race condition

We found in the sequins log most zookeeper errors happen because of 'node does not exist'. TriggerCleanup() is used to cleanup the old znodes for db versions that sequins nodes are no longer serving. But it takes a naive approach to deletes znodes. It just traverses the directory tree and delete the non-ephemeral znodes that can be deleted.  So there is a race condition when a znode for a new db version was just created and no partition of the db version has been downloaded by any sequins node (therefore no ephemeral nodes created in the db version znode). TriggerCleanup() can delete the db version znode right after it is created, causing all sequins nodes watching the db version znode to get 'node does not exist' error and they have to reconnect to zookeeper. The reconnecting to zookeeper is very expensive in sequins. It also causes unnecessary load on zookeeper server. We should avoid this race condition.
There is another problem with TriggerCleanup(). It launches a large number of goroutines to attempt to delete znodes in parallel. We found this behavior causes scheduling delay for other goroutines. It causes the 504 errors because the goroutines serving http requests are timed out. We also need to change this behavior.
r? @bartle-stripe @vasi-stripe

lgtm

lgtm


Backgrounder plugin still failing

Im running the latest 4.2 release against logs generated from TS 2020.2.3 (Build 2020-6-26) and still generating errors when running the backgrounder logs. This issue was noted as fixed in the latest release.
Reported by: 'BackgrounderPlugin'. Error: 'Failed to process string as Backgrounder event. This is expected for logs prior to 10.4'

Hi Paul,
The message you see and the note in the release notes are different things (the note in the release notes is not very specific unfortunately).
The issue we fixed in v4.2 (iirc) is that LogShark wasn't throwing any errors, but produced very little (if any) output with the Backgrounder plugin. This happened because log message format changed and rules Backgrounder plugin had were ignoring the lines they were supposed to parse.
The error message you see simply means that LogShark wasn't able to parse a particular line in Backgrounder logs at all. This happens quite often because Backgrounder sometimes logs two messages on the same line. The message is generally benign and only means that LogShark skipped that one particular line while processing. Unless you see hundreds of those messages in the run summary, feel free to ignore these messages.

thanks for that clarity but that would mean my problem is different then as the resulting output from my logs is completely empty (there is nothing getting parsed into the backgrounder output at all). I would expect there to be some output as the logs do have content in them.

Plz post a copy of Run Summary here (feel free to mask any parts that you're not comfortable sharing, i.e. log set name)

What are you looking for in the run summary? This?
      -----Data Writers Statistics-----       Lines persisted per data set:       Apache_ApacheRequests                             : 1901834 lines persisted       Backgrounder_BackgrounderJobErrors                : 274 lines persisted       ClusterController_ClusterControllerDiskIoSamples  : 41730 lines persisted       ClusterController_ClusterControllerErrors         : 53 lines persisted       ClusterController_ZookeeperFsyncLatencies         : 4 lines persisted       Config_ConfigEntries                              : 3970 lines persisted       Config_ProcessTopology                            : 24 lines persisted       Filestore_Filestore                               : 50599 lines persisted       Hyper_HyperErrors                                 : 483 lines persisted       Hyper_HyperQueries                                : 4066960 lines persisted       Netstat_NetstatEntries                            : 4221 lines persisted       Postgres_PostgresEvents                           : 242 lines persisted       ResourceManager_ResourceManagerActions            : 21 lines persisted       ResourceManager_ResourceManagerCpuSamples         : 234852 lines persisted       ResourceManager_ResourceManagerMemorySamples      : 234864 lines persisted       ResourceManager_ResourceManagerThresholds         : 32328 lines persisted       SearchServer_SearchServerEvents                   : 6888990 lines persisted       Vizportal_VizportalEvents                         : 658705 lines persisted

Scratch this. The issue was I had the old version in the env PATH so each run was using the old version rather than the new version. Run the right version and no problems

Thank you for the update - glad to hear the issue is resolved now!


Docker image fails to start with the latest version.

🐛 Bug Report
All of a sudden, a previously functional 0.7.9 image stopped working, and gets stuck starting in an endless chown loop. The image starts up, gets to "Setting data directory permissions", and then a whole bunch of chown errors arise.
To Reproduce
I am unsure what caused the problem as it was working yesterday and stopped working when I got home today.
Some additional info:
I am running this image inside docker on a Synology NAS, and I set up a user jail so I could allow the GM access to the git repo that hosts our game.
Expected behavior
I expected the image to load as usual. A work-around was provided on Foundry Discord, setting the image to 0.7.8 and the FOUNDRY_VERSION configuration variable to 0.7.9 which worked for me.
Any helpful log output
Please run this command:
docker inspect --format='{{range $k, $v := .Config.Labels}}
    {{- printf "%s = \"%s\"\n" $k $v -}} {{end}}' \
    felddy/foundryvtt:latest

Paste the results here:
com.foundryvtt.version = "0.7.9"
org.opencontainers.image.authors =<EMAIL_ADDRESS>org.opencontainers.image.created = "2020-12-19T18:22:45Z"
org.opencontainers.image.description = "An easy-to-deploy Dockerized Foundry Virtual Tabletop server."
org.opencontainers.image.licenses = "MIT"
org.opencontainers.image.revision = "d7bbea8063e5cbf8cc93ef114aca843be45d299d"
org.opencontainers.image.source = "https://github.com/felddy/foundryvtt-docker.git"
org.opencontainers.image.title = "foundryvtt-docker"
org.opencontainers.image.url = "https://github.com/felddy/foundryvtt-docker"
org.opencontainers.image.vendor = "Geekpad"
org.opencontainers.image.version = "0.7.9"

Run the container with CONTAINER_VERBOSE set to true,and paste any useful
log output here:
Entrypoint | 2020-12-27 22:01:45 | [debug] Timezone set to: UTC
Entrypoint | 2020-12-27 22:01:45 | [info] Starting felddy/foundryvtt container v0.7.9
Entrypoint | 2020-12-27 22:01:45 | [debug] CONTAINER_VERBOSE set.  Debug logging enabled.
Entrypoint | 2020-12-27 22:01:45 | [info] Reading configured secrets from: /run/secrets/config.json
Entrypoint | 2020-12-27 22:01:46 | [info] Foundry Virtual Tabletop 0.7.9 is installed.
Entrypoint | 2020-12-27 22:01:46 | [info] Not modifying existing installation license key.
Entrypoint | 2020-12-27 22:01:46 | [info] Setting data directory permissions.
chown: /data/lib_all/i686: No such file or directory
chown: /data/lib_all/libsynotrigger.so: No such file or directory
chown: /data/lib_all/perl5: No such file or directory
chown: /data/lib_all/sharing/plugin/SYNO.SDS.App.SharingUpload.Application.so: No such file or directory
chown: /data/lib_all/sharing/plugin/SYNO.SDS.App.FileStation3.Instance.so: No such file or directory
chown: /data/lib_all/ld-linux.so.2: No such file or directory
chown: /data/lib_all/terminfo: No such file or directory
chown: /data/Data/var/packages/git/etc: No such file or directory
chown: /data/Data/var/packages/git/target: No such file or directory
chown: /data/Data/etc/ssl/certs/Swisscom_Root_CA_1.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/c9f83a1c.0: No such file or directory
chown: /data/Data/etc/ssl/certs/WellsSecure_Public_Root_Certificate_Authority.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/DST_ACES_CA_X6.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/157753a5.0: No such file or directory
chown: /data/Data/etc/ssl/certs/NetLock_Express_=Class_C=_Root.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/UTN_USERFirst_Hardware_Root_CA.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/3efd4dc0.0: No such file or directory
chown: /data/Data/etc/ssl/certs/Equifax_Secure_eBusiness_CA_1.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/7d5a75e4.0: No such file or directory
chown: /data/Data/etc/ssl/certs/NetLock_Qualified_=Class_QA=_Root.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/StartCom_Certification_Authority_2.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/S-TRUST_Authentication_and_Encryption_Root_CA_2005_PN.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/9f0f5fd6.0: No such file or directory
chown: /data/Data/etc/ssl/certs/cb357862.0: No such file or directory
chown: /data/Data/etc/ssl/certs/Certinomis_-_Autorité_Racine.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/cbeee9e2.0: No such file or directory
chown: /data/Data/etc/ssl/certs/StartCom_Certification_Authority_G2.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/Visa_eCommerce_Root.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/0810ba98.0: No such file or directory
chown: /data/Data/etc/ssl/certs/Equifax_Secure_Global_eBusiness_CA.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/56657bde.0: No such file or directory
chown: /data/Data/etc/ssl/certs/876f1e28.0: No such file or directory
chown: /data/Data/etc/ssl/certs/TURKTRUST_Certificate_Services_Provider_Root_2007.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/TC_TrustCenter_Class_3_CA_II.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/b6c5745d.0: No such file or directory
chown: /data/Data/etc/ssl/certs/Comodo_Secure_Services_root.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/3ee7e181.0: No such file or directory
chown: /data/Data/etc/ssl/certs/Comodo_Trusted_Services_root.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/1874d4aa.0: No such file or directory
chown: /data/Data/etc/ssl/certs/f38a011e.0: No such file or directory
chown: /data/Data/etc/ssl/certs/Swisscom_Root_CA_2.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/Swisscom_Root_EV_CA_2.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/9007ae68.0: No such file or directory
chown: /data/Data/etc/ssl/certs/9d520b32.0: No such file or directory
chown: /data/Data/etc/ssl/certs/Equifax_Secure_CA.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/Certification_Authority_of_WoSign_G2.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/WoSign_China.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/5620c4aa.0: No such file or directory
chown: /data/Data/etc/ssl/certs/c99398f3.0: No such file or directory
chown: /data/Data/etc/ssl/certs/China_Internet_Network_Information_Center_EV_Certificates_Root.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/c5e082db.0: No such file or directory
chown: /data/Data/etc/ssl/certs/790a7190.0: No such file or directory
chown: /data/Data/etc/ssl/certs/bb2d49a0.0: No such file or directory
chown: /data/Data/etc/ssl/certs/ef2f636c.0: No such file or directory
chown: /data/Data/etc/ssl/certs/IGC_A.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/Staat_der_Nederlanden_Root_CA.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/cfa1c2ee.0: No such file or directory
chown: /data/Data/etc/ssl/certs/CA_WoSign_ECC_Root.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/Verisign_Class_1_Public_Primary_Certification_Authority_-_G2.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/861e0100.0: No such file or directory
chown: /data/Data/etc/ssl/certs/AddTrust_External_Root.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/c679bc3f.0: No such file or directory
chown: /data/Data/etc/ssl/certs/UTN_USERFirst_Email_Root_CA.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/034868d6.0: No such file or directory
chown: /data/Data/etc/ssl/certs/ACEDICOM_Root.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/e536d871.0: No such file or directory
chown: /data/Data/etc/ssl/certs/024dc131.0: No such file or directory
chown: /data/Data/etc/ssl/certs/8b59b1ad.0: No such file or directory
chown: /data/Data/etc/ssl/certs/TÜBİTAK_UEKAE_Kök_Sertifika_Hizmet_Sağlayıcısı_-_Sürüm_3.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/a760e1bd.0: No such file or directory
chown: /data/Data/etc/ssl/certs/65b876bd.0: No such file or directory
chown: /data/Data/etc/ssl/certs/Security_Communication_EV_RootCA1.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/PSCProcert.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/c5d3212a.0: No such file or directory
chown: /data/Data/etc/ssl/certs/57bbd831.0: No such file or directory
chown: /data/Data/etc/ssl/certs/ae8153b9.1: No such file or directory
chown: /data/Data/etc/ssl/certs/f060240e.0: No such file or directory
chown: /data/Data/etc/ssl/certs/0d1b923b.0: No such file or directory
chown: /data/Data/etc/ssl/certs/Root_CA_Generalitat_Valenciana.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/CA_Disig_Root_R1.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/7992b8bb.0: No such file or directory
chown: /data/Data/etc/ssl/certs/Verisign_Class_3_Public_Primary_Certification_Authority_-_G2.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/415660c1.1: No such file or directory
chown: /data/Data/etc/ssl/certs/2ab3b959.0: No such file or directory
chown: /data/Data/etc/ssl/certs/RSA_Security_2048_v3.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/Buypass_Class_2_CA_1.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/Verisign_Class_3_Public_Primary_Certification_Authority_2.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/bd1910d4.0: No such file or directory
chown: /data/Data/etc/ssl/certs/1ec4d31a.0: No such file or directory
chown: /data/Data/etc/ssl/certs/381ce4dd.0: No such file or directory
chown: /data/Data/etc/ssl/certs/Verisign_Class_2_Public_Primary_Certification_Authority_-_G2.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/CNNIC_ROOT.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/Sonera_Class_1_Root_CA.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/NetLock_Business_=Class_B=_Root.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/EBG_Elektronik_Sertifika_Hizmet_Sağlayıcısı.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/8096d0a9.0: No such file or directory
chown: /data/Data/etc/ssl/certs/24ad0b63.0: No such file or directory
chown: /data/Data/etc/ssl/certs/Juur-SK.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/26eaad2f.0: No such file or directory
chown: /data/Data/etc/ssl/certs/NetLock_Notary_=Class_A=_Root.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/CA_Disig.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/TÜRKTRUST_Elektronik_Sertifika_Hizmet_Sağlayıcısı_H6.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/StartCom_Certification_Authority.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/AC_Raíz_Certicámara_S.A..pem: No such file or directory
chown: /data/Data/etc/ssl/certs/Microsec_e-Szigno_Root_CA.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/578d5c04.0: No such file or directory
chown: /data/Data/etc/ssl/certs/WoSign.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/79ad8b43.0: No such file or directory
chown: /data/Data/etc/ssl/certs/b13cc6df.0: No such file or directory
chown: /data/Data/etc/ssl/certs/b8e83700.0: No such file or directory
chown: /data/Data/etc/ssl/certs/667c66d4.0: No such file or directory
chown: /data/Data/etc/ssl/certs/TÜRKTRUST_Elektronik_Sertifika_Hizmet_Sağlayıcısı_H5.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/3b2716e5.0: No such file or directory
chown: /data/Data/etc/ssl/certs/Verisign_Class_3_Public_Primary_Certification_Authority.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/Certplus_Class_2_Primary_CA.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/Verisign_Class_1_Public_Primary_Certification_Authority.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/415660c1.0: No such file or directory
chown: /data/Data/etc/ssl/certs/Certinomis_-_Root_CA.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/Deutsche_Telekom_Root_CA_2.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/fcac10e3.0: No such file or directory
chown: /data/Data/etc/ssl/certs/b7e7231a.0: No such file or directory
chown: /data/Data/etc/ssl/certs/6f2c1157.0: No such file or directory
chown: /data/Data/etc/ssl/certs/592c0a9a.0: No such file or directory
chown: /data/Data/etc/ssl/certs/b42ff584.0: No such file or directory
chown: /data/Data/etc/ssl/certs/812e17de.0: No such file or directory
chown: /data/Data/etc/ssl/certs/19c1fa33.0: No such file or directory
chown: /data/Data/etc/ssl/certs/d9d12c58.0: No such file or directory
chown: /data/Data/etc/ssl/certs/AddTrust_Qualified_Certificates_Root.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/67d559d1.0: No such file or directory
chown: /data/Data/etc/ssl/certs/ApplicationCA_-_Japanese_Government.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/ComSign_CA.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/AddTrust_Public_Services_Root.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/ae8153b9.0: No such file or directory
chown: /data/Data/etc/ssl/certs/5d63b0ae.0: No such file or directory
chown: /data/Data/etc/ssl/certs/S-TRUST_Universal_Root_CA.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/GeoTrust_Global_CA_2.pem: No such file or directory
chown: /data/Data/etc/ssl/certs/d957f522.0: No such file or directory

This list of files is a bit concerning.  I think you may have misconfigured your volumes.   Specifically you shouldn't have etc under Data, nor should the lib_all tree be in there.
Could you please post a list of the volumes that you have mapped into the container.

Yes.  I think the chroot jail is causing the permission errors you are seeing.  Your idea to clean up the volume is a good one.  If that doesn't fix your error please feel free to reopen this issue.

Yes.  I think the chroot jail is causing the permission errors you are seeing.  Your idea to clean up the volume is a good one.  If that doesn't fix your error please feel free to reopen this issue.


Questions about serverless.yml

Hi, AWS is quite new to me. Been struggling a lot with running Lighthouse as a Google Cloud Function.
Now that's out of the picture, but trying out your solution is very exciting for me.
However, I have some questions about this part on your blog post:
Here are the relevant parts of my serverless.yml file:
That link to the blob-commit isn't working.
I'd gladly wonder how set up serverless.yml
Could you post a more detailed serverless.yml or in better detail describe the step when you wrote:

Notice I’m putting the Lambda behind an SQS queue to throttle it. I’m using a plugin to make it easier to create the SQS queue.

Cheers!

Sorry for the late reply, I didn't saw the issue until now.
I just realised the link it to a private repository.
Actually it should be easier to set up Lighthouse in Google Cloud Functions as they support chrome out of the box.
A lot has changed since I wrote the post and the layer now works with Node 10.
I'll try to update the post soon to reflect the changes and also post a new example with a public repository.


matrix math library suggestion

Hi
Not sure if you have checked this out before but there is a matrix math library available called sb-cga.
While you own implementation are more than likely spot on the sb-cga library has the interesting property of making use of some low level SSE optimization when used with SBCL.
The library is available here https://github.com/nikodemus/sb-cga and via quicklisp.
It does have some draw backs though; it supports only vec3 and mat4x4 and only floats.
But might be worth checking out if you are interested in low level optimisation.
Cheers,

rtg-math has owned the vector/matrix code for a while. Whilst it doesnt support everything it does have an api I prefer to sb-cga.


Ec2instance watcher issue798

Fix for the issue https://github.com/Netflix/security_monkey/issues/798
Bug: Instance count information was getting incorrectly recorded
Problem: After retrieving instances using AWS API, instances list was iterated twice in loop. This lead to duplicating results and missing some other ones

Coverage decreased (-0.006%) to 63.303% when pulling 573f06990d04031beef54f6c0e768a65b090d9c4 on shrikant0013:ec2instance-watcher-issue798 into 0f29754fcc300db912a263620fa4ef1f98241ee9 on Netflix:develop.


observer: fail when requesting empty filename for observing

The observer should never accept to observe an empty file. If it does, it is a programming error and we should panic.
/cc @sttts

/approve


Improve projection detection to make Vector indexing O(1)

This was originally discovered in #104, and there's also a workaround there.
The ReturnDirection pass detects when it's safe for a function that returns something of unknown type to return it by (derived) pointer instead of by copying to an out parameter. For some reason this doesn't kick in for vector indexing. My hunch is that this is causing us to generate code that LLVM's optimiser doesn't easily see through, meaning that we don't get the constant-time goodness we would want.
This needs to be investigated and fixed if possible.

Have you considred Vector indexed by Bin n rather than Nat?
Here's a (surely invalid) sixten code for it.
-- Bin n represents n bits.
-- I (O (I H)) = 101(2)
type Bin: Nat -> Type where
  H: Bin 0
  I: Bin n -> Bin (Sunn c)
  O: Bin n -> Bin (Sunn c)

shift: n -> Bin m -> Bin (n + m)
shift Zero x = x
shift (Succ n) H = O (shift n H)
shift (Succ n) (I x) = I (shift n x)
shift (Succ n) (O x) = O (shift n x)

Vector : forall n. Bin n -> Type -> Type
Vector (H) _ = Unit
Vector (I H) a = a
Vector @n (I x) a = Pair (Pair (Vector y a) (Vector y a)) (Vector x a)
  where y = I (shift I (n - 1))
Vector (O x) a = Vector x a)

index : forall a. forall n. (len : Bin n) -> Vector len a -> Bin n -> Maybe a
index H MkUnit _ = Nothing
index (I H) x 0 = Just x
index (I y) (MkPair x _) (I z) = index y x z
index (I y) (MkPair _ x) (O z) = index y x z
index (O y) x (O z) = index y x z

With this definition, Vector n a is built like a binary trie and an index can be considered to be a path of trie .
I think this definition represents the machine behavior more precisely. Random access is not O(1) but O(log n) where n is the length of address. Roughly speaking, memory access in 64bit address is 2 time slower than in 32 bit one.
I'm not sure LLVM can generate the optimized binary from this definition.
But I think this definition gives sufficient information to do so.
Because the pattern maching process within index skips elements in MkPairs whose summation of size is equal to given index i.

I haven't tried that, but it sounds like a fun idea to try!


Profiling: Configure Reactotron with project.

Cool. Do we want to enable the integration automatically during all dev modes?
This uncommenting is fine though (and safe too).

Not sure...If we enable it in DEV then, debugging with Reactotron will be quicker. But, if redux logger is also enabled, then general slowness might be observed.
Also, devs need to install the Reactotron desktop application and reload the app to see the logs, so if they aren't aware of this, then the logs are not being used.


Create a field-existence validator plugin

Create a field-existence validator plugin. This validator plugin verifies that all fields on a definition are defined on the schema for that definition.
For example, this validator would fail on the following definition because it has badfield, which is not defined on the spec definition for model:
model:
  name: Model with erroneous field
  badfield: This shouldn't be allowed

AC:

[ ] The validator plugin implements the following plugin hooks

[ ] get_plugin_aac_definitions()
[ ] register_validators()

[ ] The validator contains acceptance scenarios for

[ ]  validation success
[ ]  validation failure

[ ] Validation fails if a definition has a field that is not defined in it's schema definition
[ ] The validation is tied to the enum definition

Believe this was met with a legacy validator of defined_fields and exists within the 0.4.0 baseline of PR #714 that has a global schema constraint evaluating for defined fields of a model.


refactor: remove unused WebUI field from NodeConfig

Its not used so nixing it.

I've removed it over in #3132 although there is some strange behaviour with the webui configflags :|

Made redundant by #3132.


Fix permission for the docker socket in datadog

@pierreozoux can you please bump the chart version?

@lachie83 done

/ok-to-test


App crashes when receiving Push (FCM)

Hi,
since FCM is "the future" I think we need a solution for this, so we can work with it :)
If i send a test notification over FCM my app crashes.
[ERROR] :  TiApplication: (AsyncTask #3) [56387,57328] Sending event: exception on thread: AsyncTask #3 msg:java.lang.NullPointerException: Attempt to invoke virtual method 'boolean com.google.gson.JsonObject.has(java.lang.String)' on a null object reference; Titanium 6.0.1,2016/12/19 16:51,undefined [ERROR] :  TiApplication: java.lang.NullPointerException: Attempt to invoke virtual method 'boolean com.google.gson.JsonObject.has(java.lang.String)' on a null object reference [ERROR] :  TiApplication: 	at ti.goosh.IntentService.parseNotification(IntentService.java:122) [ERROR] :  TiApplication: 	at ti.goosh.IntentService.onMessageReceived(IntentService.java:54) [ERROR] :  TiApplication: 	at com.google.android.gms.gcm.GcmListenerService.zzq(Unknown Source) [ERROR] :  TiApplication: 	at com.google.android.gms.gcm.GcmListenerService.zzp(Unknown Source) [ERROR] :  TiApplication: 	at com.google.android.gms.gcm.GcmListenerService.zzo(Unknown Source) [ERROR] :  TiApplication: 	at com.google.android.gms.gcm.GcmListenerService.zza(Unknown Source) [ERROR] :  TiApplication: 	at com.google.android.gms.gcm.GcmListenerService$1.run(Unknown Source) [ERROR] :  TiApplication: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112) [ERROR] :  TiApplication: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
Not sure how to set the "alert" object, since there is no such input on the fcm page. Just Keypairs. Any help would be great.

Try a payload as in the example I just committed here https://github.com/caffeinalab/ti.goosh/commit/37c9ce7cc0686c7fdc2fffb3ff248a3f76223c85
The module expects a data property and all the other properties to be in it therefore if you don't set it the app crashes because it tries to read data as an object. Firebase supports the data property so as long as you use it this way it works. You can keep using the default Firebase properties such as title but they're not considered at all by the module.


Discrepancy between the LD_PRELOAD hack and the curl script

I noticed that the rss-bridge, which I enabled to use curl-impersonate via LD_PRELOAD hack, no longer impersonates properly.
I confirmed that with your upstream docker image with:
$ docker run -it --rm lwthiker/curl-impersonate:0.5-ff-slim-buster bash
> export LD_PRELOAD=/usr/local/lib/libcurl-impersonate-ff.so
> export CURL_IMPERSONATE=ff100
> apt install curl
> curl "https://allegro.pl/strefaokazji"
<title>allegro.pl</title>
Please enable JS and disable any ad blocker

> curl_ff100 "https://allegro.pl/strefaokazji"
> (returns valid HTML)

It worked just fine previously, so I am not sure if it regressed due to some underlying debian image updates (just checked, Alpine is also affected), or the impersonation is somehow flawed in LD_PRELOAD mode, and the tightening on allegro.pl end eventually caught it.
EDIT: also checked with curl-impersonate:0.5-chrome, same problem.

The LD_PRELOAD method does not work with stock curl. This is because curl overrides the settings set by libcurl-impersonate after they are set. This behavior is documented on the README:

Note that the LD_PRELOAD method will NOT WORK for curl itself because the curl tool overrides the TLS settings. Use the wrapper scripts instead.

There is not much to do about it, as we don't have control over the behavior of the upstream curl.

There is not much to do about it, as we don't have control over the behavior of the upstream curl.

Oh well, my bad! Somehow I was under impression that standalone curl worked previously. That having said, the rss-bridge uses libcurl, so it should work there just fine. And  things got back to normal yesterday such that I am no longer getting the 403s. I don't know how to explain that, to be honest. I will close this for now and if this happens again, I'll investigate and reopen.

@lwthiker I am getting this issue again and it is a
Duplicate of #147

@lwthiker I am getting this issue again and it is a
Duplicate of https://github.com/lwthiker/curl-impersonate/issues/147


feat: Update API stack to register SageMaker Endpoints

Decouple the need to have specific Register call in the deployment pipeline, by adding a EventBridge rule to trigger when endpoint is created/updated/deleted.
Only register events that have been tagged with ab-testing:enabled = true by CodePipeline.  Also tag properties for strategy, epsilon and warmup.

Closed with PR merge.


Api

Also this PR is quite big of size and can be broken down in to 2 PRs (auth/maps). Smaller PRs are preferred since it’s easier to review and rollback.