Actually use dbg config in win RBE debug jobs

I'm looking to resolve the
<EMAIL_ADDRESS>error while loading shared libraries: ucrtbased.dll: cannot open shared object file: No such file or directory problem. Might be due to some problems with our docker image.


Starbucks clone

Prepared a clone of everyone's favourite Starbucks! Used html, css. Issue no. #6. Let me know if any changes are required.

Sure... I'll do changes today till evening.


Add Github API + GraphQL Repository Fetching

Motivation

To make our Leaderboard possible, we first have to fetch the required PRs / Comments data from Github's API and store them in the database, as well as make the data available through our own API.
Description

Adds application-local.yml-support for the Github PAT
Adds the database structure for Github's API/Webhook schema
Provides a sample runner to store Hephaestus`s data in the database
Provides a Github API Client + GraphQL Client as fallback

Next up for the leaderboard (in other PRs):

Webhook integrations (NATS)
More endpoints depending on which data we need in our front-end

Screenshots (if applicable)

Checklist
General

[x] PR title is clear and descriptive
[x] PR description explains the purpose and changes
[x] Code follows project coding standards
[x] Self-review of the code has been done
[x] Changes have been tested locally
[ ] Screenshots have been attached (if applicable)
[x] Documentation has been updated (if applicable)

Server (if applicable)

[x] Code is performant and follows best practices
[x] No security vulnerabilities introduced
[x] Proper error handling has been implemented
[ ] Added tests for new functionality
[ ] Changes have been tested in different environments (if applicable)

I think the code looks good now, maybe you can take a quick look at ResponseEntity and the returned NotFoundException. Also you need to update the angular API client library, there is a command that you need to run in package.json. Afterwards we can merge this PR :)


Test/add e2e stage

AB#5063
This PR adds a new stage that reports stage-level results to Aria/Kusto.

I don't see any obvious issues with the changes to the file, but I think the real verification we want from this is that telemetry actually made it to Kusto.


[http-fault-injector] Target net7.0 to enable HTTPS on Mac with no keychain popup

Fixed in https://github.com/dotnet/aspnetcore/issues/41879

@weshaggard: Are you OK with http-fault-injector targeting net7.0?  I know we are trying to make all our tools consistent and target net6.0, but the easiest way to avoid this problem is upgrading to net7.0.  http-fault-injector doesn't have many users, and I'm fine with forcing them to use .NET 7.

I don't have a concern if we have a good reason. Do we have people hitting this issue that need this fix? If so will we have this issue in other tools like Test Proxy?

Alan is trying to use http-fault-injector in a pipeline that runs on MacOS.  I will need to investigate why test-proxy isn't hitting this on Mac.  There might be a slight difference in the tool code itself, or in how the tools are being launched.

@weshaggard: Are you OK with http-fault-injector targeting net7.0? I know we are trying to make all our tools consistent and target net6.0, but the easiest way to avoid this problem is upgrading to net7.0. http-fault-injector doesn't have many users, and I'm fine with forcing them to use .NET 7.

We should aim to publish standalone executables for all our tools, then we don't have this problem where users are forced to have the right NET installed. It's a usability nightmare getting 6 and 7 installed side by side on linux, for example.

@weshaggard: Are you OK with http-fault-injector targeting net7.0? I know we are trying to make all our tools consistent and target net6.0, but the easiest way to avoid this problem is upgrading to net7.0. http-fault-injector doesn't have many users, and I'm fine with forcing them to use .NET 7.

We should aim to publish standalone executables for all our tools, then we don't have this problem where users are forced to have the right NET installed. It's a usability nightmare getting 6 and 7 installed side by side on linux, for example.

@benbp: Do you know what needs to change in HttpFaultInjector.csproj to enable this?  Do we have existing examples of publishing standalone executables in the tools repo?

@mikeharder here is how proxy does it. @scbedd did a lot of work to get our tools dotnet pipeline able to sign mac binaries, etc.
https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/ci.yml#L52-L68

@mikeharder here is how proxy does it. @scbedd did a lot of work to get our tools dotnet pipeline able to sign mac binaries, etc.
https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/ci.yml#L52-L68
[edit] in the csproj, probably these: https://github.com/Azure/azure-sdk-tools/blob/main/tools/test-proxy/Azure.Sdk.Tools.TestProxy/Azure.Sdk.Tools.TestProxy.csproj#L6-L8

To be clear though, when you package for standalone the executable becomes the actual namespace. so Azure.Sdk.Tools.TestProxy is the standalone executable regardless of <packastool> or not.
I agree with the suggestion to just add the standalone matrix. That should "just work" ™


layers/meta-opentrons: binding for opentrons cli

Add explicit executable shell scripts for opentrons_simulate and opentrons_execute.
Normally, when you install python packages that have script entrypoints, the install process will generate the scripts and put them in the appropriate place. Here, we have a separate python environment in a special directory that the system doesn't know about - it's used by setting environment variables in systemd units. That means the installation won't work. So we do it ourself here.
Closes RESC-285

Can you explain how this fixes the issue and then in what context these 2 files will be run?
Why are we fixing it here and not in the package?

Can you explain how this fixes the issue and then in what context these 2 files will be run? Why are we fixing it here and not in the package?

updated the pr description


Menu items wrongly displayed

Checklist

[X] I have read and accepted the contributing guidelines.

How did you create the site?
Generated from chirpy-starter
Describe the bug
In the menu bar of https://chirpy.cotes.page, and in my local installation, the menu items are not displayed properly.
For example, have a look at "Archive" and "About" in the menu at the left of the following screenshot:

When you move the mouse over the text, it gets fixed. But when you leave, some get broken again.
Steps To Reproduce

Go to https://chirpy.cotes.page
Look at "Archive" and "About" in the left menu bar

Expected Behavior
The text should be clear to read.
Environment
On https://chirpy.cotes.page you can reproduce the error.
In local too:

Ruby: 3.2.2
Jekyll: 4.3.2
Chirpy: jekyll-theme-chirpy (6.2.2)

Anything else?
Thanks!

The problem seems to be my version of Chrome. In Firefox, and other Chrome browsers, it works.

I'm reopening this as it's still an issue.
Chrome version 117.0.5938.92
Ubuntu 23.04
I disabled the extensions and tried in incognito mode.
Someone else reported the same problem too. Although it's ok for others.
Removing the body font-family "Source Sans Pro","Microsoft Yahei",sans-serif;` looks right.
It could be an issue with Ubuntu default fonts — which I haven't changed.

You might consider raising an issue with Chrome.

Thanks @cotes2020.
Meanwhile, in order to replace the font family in my blog, how could I redefine $font-family-base from chirpy's _sass/addon/variables.scss in my blog? Thanks!

@cotes2020 I tried that font family in a simple html file [1] and it's displayed properly in Chrome. Yet, my Chirpy site and https://chirpy.cotes.page don't.
By the way, even if it were a problem of Chrome, it would still be an issue for everyone with old Chrome versions.
So it would be great to have a way to redefine $font-family-base in my project.
I tried to redefine variables.scss but it didn't work.
Should it be placed in the directory sass, _sass or sass/addons?
What do I need to add to _config.yml under sass:?
Thanks!
[1]
<!DOCTYPE html>

test

test

test

https://github.com/cotes2020/jekyll-theme-chirpy/discussions/1185

The new version of Chrome (118) fixes it.
It seems that the last version of Chromium in deb repositories still fails, but I guess it will be fixed soon too.


Fails to validate with pod lib lint on Xcode 7.1 and later

 -> CRToastSwift (0.1.0)
    - ERROR | [iOS] xcodebuild: Returned an unsuccessful exit code. You can use `--verbose` for more information.
    - NOTE  | xcodebuild:  <module-includes>:1:9: note: in file included from <module-includes>:1:
    - NOTE  | xcodebuild:  Target Support Files/CRToast/CRToast-umbrella.h:3:9: note: in file included from Target Support Files/CRToast/CRToast-umbrella.h:3:
    - ERROR | [iOS] xcodebuild:  CRToast/CRToast/CRToast.h:16:9: error: include of non-modular header inside framework module 'CRToast.CRToast'
    - ERROR | [iOS] xcodebuild:  CRToast/CRToast/CRToast.h:17:9: error: include of non-modular header inside framework module 'CRToast.CRToast'
    - ERROR | [iOS] xcodebuild:  /Users/masahiko/Documents/Xcode/CRToastSwift/Pod/Classes/CRToastSwift.swift:27:8: error: could not build Objective-C module 'CRToast'

[!] CRToastSwift did not pass validation, due to 4 errors.
You can use the `--no-clean` option to inspect any issue.

For workaround, for the time being, we'll validate and push this pod using Xcode 7.0.1.

Probably related to CocoaPods/CocoaPods#4420

This issue seems to have been fixed by CocoaPods update.
We don't face this issue using CocoaPods beta 3 and Xcode 7.2.1


Rename osc to oc in welcome page

Sorry for this, I've forgot about this in the previous PR..

@bparees PTAL


Show contents of your m3u8.
I suppose you need to:
this.mpv.option("demuxer-lavf-o", "protocol_whitelist=[file,tcp,http,https]");

if you're trying to open local m3u8 file with http/https links. It's disabled by default for security reasons.
See https://github.com/mpv-player/mpv/issues/2937 https://github.com/mpv-player/mpv/issues/4488

@Kagami  Thank you, it's work!


Add fred data source

Codecov Report
Attention: Patch coverage is 0% with 161 lines in your changes missing coverage. Please review.

Project coverage is 73.53%. Comparing base (b39c1d2) to head (be5ca98).

Files with missing lines
Patch %
Lines

quantflow/cli/commands/fred.py
0.00%
45 Missing :warning:

quantflow/cli/commands/vault.py
0.00%
35 Missing :warning:

quantflow/data/vault.py
0.00%
30 Missing :warning:

quantflow/cli/commands/stocks.py
0.00%
22 Missing :warning:

quantflow/cli/app.py
0.00%
21 Missing :warning:

quantflow/data/fred.py
0.00%
7 Missing :warning:

quantflow/cli/settings.py
0.00%
1 Missing :warning:

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #11      +/-   ##
==========================================
- Coverage   78.66%   73.53%   -5.13%
==========================================
  Files          35       38       +3
  Lines        1964     2101     +137
==========================================
  Hits         1545     1545
- Misses        419      556     +137

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.


[Bug Report] No logs found in rotating file when used in an async sink (using log crate proxy)

Describe the bug
When running an async sink using a rotating file sink and a stderr sink, I am able to find all the logs correctly on the stderr, but none of them seem to be redirected to the rotating file sink. Note: I am using the log crate proxy to redirect application logs sent to the log crate.
To Reproduce
Steps to reproduce the behavior:

Setup a stderr sink

    let stderr_sink = spdlog::sink::StdStreamSink::builder()
        .std_stream(spdlog::sink::StdStream::Stderr)
        .level_filter(spdlog::LevelFilter::All)
        .build()
        .with_context(|| "Failed to build stderr sink")?;

Setup a rotating sink

    let rotating_sink = spdlog::sink::RotatingFileSink::builder()
        .base_path(base_file)
        .rotation_policy(spdlog::sink::RotationPolicy::Daily { hour: 0, minute: 0 })
        .max_files(10)
        .level_filter(spdlog::LevelFilter::All)
        .build()
        .with_context(|| "Failed to build rotating file sink")?;

Setup async sink

    let rotating_sink = Arc::new(rotating_sink);
    let stderr_sink = Arc::new(stderr_sink);
    let async_sink = AsyncPoolSink::builder()
        .sink(stderr_sink.clone())
        .sink(rotating_sink.clone())
        .level_filter(spdlog::LevelFilter::All)
        .overflow_policy(spdlog::sink::OverflowPolicy::Block)
        .build()
        .with_context(|| "Failed to build async pool sink")?;

Setup logger

    let logger = spdlog::Logger::builder()
        .sink(Arc::new(async_sink))
        .level_filter(spdlog::LevelFilter::All)
        .flush_level_filter(spdlog::LevelFilter::MoreSevereEqual(spdlog::Level::Warn))
        .build()
        .with_context(|| "Failed to build logger")?;

Setup log crate proxy

    log::set_max_level(log::LevelFilter::Trace);

    //Connect to log crate
    spdlog::init_log_crate_proxy()
        .expect("users should only call `init_log_crate_proxy` function once");

    spdlog::log_crate_proxy().set_logger(Some(logger.into()));

Expected behavior
File should also have the same logs as the stderr stream
Screenshots

Environments

OS: Windows 11
Version: Build 22631

Additional context
I am using the log crate proxy mechanism

Hi, @SpriteOvO; thanks so much for the speedy and detailed answer.

You can try setting up an automatic periodic flush via Logger::set_flush_period, which will ensure that the logs are flushed into the file at the regular interval.

The Logger::set_flush_period worked. Most of my logs were info level or below, so they never got forwarded.
Thanks a lot

I recommend reading discussion https://github.com/SpriteOvO/spdlog-rs/discussions/57 first so you will understand how they work internally

I did go through the ASyncPoolSink documentation. Personally,  I was concerned about the stderr logger more than the file based one. If I understand correctly, the stderr in rust is unbuffered. So even inside of a buffer, it might block my operations. That is what I am primarily trying to avoid.
My application generates about  1 log/5ms. I am not sure if the stderr stream might be able to handle that in a way that does not impact performance. Which is why I was offloading that to the AsyncSink to forward it to a separate thread altogether.


How to interface with the same method?

How to interface with the same method?
RIDL!(#[uuid(0x4d93059d, 0x097b, 0x4651, 0x9a, 0x60, 0xf0, 0xf2, 0x51, 0x16, 0xe2, 0xf3)]
interface IDCompositionVisual(IDCompositionVisualVtbl): IUnknown(IUnknownVtbl) {
    fn SetOffsetX(
        offsetX: c_float
    ) -> HRESULT,
    fn SetOffsetX(
        animation: *const IDCompositionAnimation
    ) -> HRESULT,
  ...

→error[E0124]: field SetOffsetX is already declared

The thing about COM interfaces is that the names themselves don't actually matter. The only thing that matters is the uuid, the signatures of the methods (but not their names!), and the number and order of the methods. You could easily rename them to SetOffsetX1 and SetOffsetX2 or provide other more descriptive names.

Thanks! rename seq number.


Make player-moc use filename if no song name

Not every file necessarily has the song field set to anything. In those
cases to avoid empty output, e.i. " - ", the filename is printed instead.

Yes seem to be working fine for all three cases.

Thanks!


gpt-4-vision-preview - receives an incomplete response

After sending an image, I receive an incomplete response in the “content” key
(
    [index] => 0
    [message] => OpenAI\Responses\Chat\CreateResponseMessage Object
        (
            [role] => assistant
            [content] => The image shows a Nikon DSLR camera with a telephoto lens attached, resting
            [toolCalls] => Array
                (
                )
            [functionCall] =>
        )
    [finishReason] =>
)

Why is the answer text cut?

I think your 'max_tokens' is too low.
Increase the 'max_tokens' value to 900 (for example) like this: 'max_tokens' => 900.

@shakhbiev I see you are getting a response. Can you share your code? I keep getting an error when sending the request with the library but not with CURL ... same request. #258

@shakhbiev I agree with @ThibautPV

@shakhbiev I agree with @ThibautPV

No. I added max_tokens=1000, but the problem is still solved. Now I checked, even without specifying max_tokens everything works, I get the full text

Perhaps the problem was on the OpenAI side

@shakhbiev I agree with @ThibautPV

Yes, problem solved!


Catch does not return Error

Why does catch return an object and not an error object?

This is how it is done with Angular's $http, which this library is based on. More or less it's an artifact and not necessarily the right way to do things. I would like to move to a more consistent error path where an Error is used that has response details attached if relevant.
function createError(message, details) {
  var err = new Error(message);

  Object.keys(details, function (key) {
    err[key] = details[key];
  });

  return err;
}

createError('timeout of ' + config.timeout + 'ms exceeded', {
  code: 'ECONNABORTED',
  config: config
});

I would also like to change the lib to only reject the Promise when an actual error occurs, regardless of HTTP status code. Whereas now anything outside the 200-300 range is rejected.

@mzabriskie
You could also consider the use of a config flag to determine what action to take when a non 2xx/3xx response is returned.
The request-promise lib uses the simple flag/option for this. When simple is true, request-promise rejects on non-network errors (2xx, 3xx). When the flag is false, it rejects nothing and becomes the user's responsibility. I've found this to be very useful and have applications that use both settings. I think I commonly have simple set to true as it is generally a better default IMO.
https://github.com/request/request-promise#get-a-rejection-only-if-the-request-failed-for-technical-reasons

Closing as duplicate of #24


ZLib The DirectStorage request failed

When I compiled in release mode and ran with a 2Gbyte input file, got below error.
What can cause this, something I need to configure in win11?
Compressing N:\Data\data.cow to N:\Data\data.cow.gdeflate in 162x16 MiB chunks
Total:<PHONE_NUMBER> --><PHONE_NUMBER> bytes (100.005%)
Compressing N:\Data\data.cow to N:\Data\data.cow.zlib in 162x16 MiB chunks
Total:<PHONE_NUMBER> --><PHONE_NUMBER> bytes (99.8229%)
Uncompressed:
16 MiB staging buffer: ..........  4.10728 GB/s mean cycle time: 370784527
32 MiB staging buffer: ..........  6.78455 GB/s mean cycle time: 308674715
64 MiB staging buffer: ..........  5.65416 GB/s mean cycle time: 339077114
128 MiB staging buffer: ..........  5.47467 GB/s mean cycle time: 363720296
256 MiB staging buffer: ..........  4.65176 GB/s mean cycle time: 405283062
512 MiB staging buffer: ..........  4.09985 GB/s mean cycle time: 448052001
1024 MiB staging buffer: ..........  4.08624 GB/s mean cycle time: 188527878
ZLib:
16 MiB staging buffer: The DirectStorage request failed! HRESULT=0x89240008
13052685   16782342
Q:\git_external\DirectStorage\Samples\GpuDecompressionBenchmark\x64\Release\GpuDecompressionBenchmark.exe (process 9040) exited with code -1073740791.
Press any key to close this window . . .

I think that this is a bug in the sample.
That error code is E_DSTORAGE_REQUEST_TOO_LARGE.  This error happens when some part of the request (either the compressed or uncompressed part) is larger than the staging buffer size.  As the sample is splitting the source data up into 16MiB chunks, the uncompressed size must be 16MiB, so my guess is that zlib compression has resulted in data that is actually larger than the source data.  This can happen - in this case the sample should really detect and handle this case.
In real-life, the right thing to do here would probably be to drop down to just storing that chunk uncompressed.
For the benchmark app, I'm not so sure since the whole point of it is to look at the performance of the codec.  Possibly the right thing to do would be to just skip the 16MiB staging buffer case when this happens.
If you want to try debugging this with your data set to see if I'm correct, have a look in GpuDecompressionBenchmark.cpp and search for queue->EnqueueRequest.  Maybe there you could add a check to validate that both request.Source.File.Size and request.UncompressedSize are less than stagingSizeMiB * 1024 * 1024 - if they're not then the request will fail and this particular test should be skipped.


Helper function to get recently updated partitions

Is your feature request related to a problem? Please describe.
For streaming systems (or batch systems that run in high frequency) that write data into delta tables, it's a common problem to have lots of small files. In many cases, it's not practical to auto compact because of various reasons, for example

Auto compaction is not available in Delta lake before 3.1.0
Auto compaction might not be well supported outside Spark

One way to solve this is to have a separate process that perform optimization regularly on these delta tables. However it's not a good idea to optimize the entire table whenever without any constraint. A few example reasons:

While in theory optimize is a no-op if the partitions weren't updated, it still takes some overhead per partition to determine it's a no-op. This could add up quite significantly when you have lots of partitions.
If the optimize operation included z-order, subsequent z-order operations won't be no-op even if the partitions weren't updated

Describe the solution you'd like
A helper function to find out which partitions have been updated between some time period, for example
def get_updated_partitions(delta_table: DeltaTable, start_time: datetime.datetime, end_time: datetime.datetime, exclude_optimize_operations: bool) -> list[dict[str, str]]

The exclude_optimize_operations flag is necessary because optimization operations themselves are also update operations. If the operations are not excluded, they might cause a feedback loop since they will keep showing up in the output.
All the information needed for this features should be available in the transaction log.
Describe alternatives you've considered
Optimizing the entire table and accept the overhead
Not sure what's a good alternative once z-order is used however
Additional context
N/A
Willingness to contribute
Would you be willing to contribute an implementation of this feature?

[x] Yes. I can contribute this feature independently.
[ ] Yes. I would be willing to contribute this feature with guidance from the mack community.
[ ] No. I cannot contribute this feature at this time.

Sounds good to me.  Only worry is that we won't be able to access the necessary information from the transaction log without using private methods.
The Delta Rust transaction log APIs expose more info.  Anyways, I am cool with this, just want to make sure we rely on public interfaces!

Sadly the transaction log information seems to only be exposed in the Scala version, not Python one :(
https://books.japila.pl/delta-lake-internals/DeltaLog/
If we wanna do this in pyspark, we would have to reach into the JVM get this information. Not sure if that would count  as public interfaces though. The delta-spark library regularly reach into the JVM from the pyspark side, for example
https://github.com/delta-io/delta/blob/master/python/delta/tables.py#L52C14-L52C18

@huynguyent - you can add this function to levi if you'd like: https://github.com/MrPowers/levi
There is a get_add_actions method in Delta Rust that probably exposes the necessary details: https://delta-io.github.io/delta-rs/api/delta_table/#deltalake.DeltaTable.get_add_actions

Yup, levi might be a more straightforward place for this feature. I'll raise an isse and look into implementing it


Update lowrisc_ibex to lowRISC/ibex@96a1c02b

Update code from upstream repository
https://github.com/lowRISC/ibex.git to revision
96a1c02ba03400a8b0d69f08b97004adc436dfb7

[dv] Increase iterations and instructions in riscv_rf_intg_test (Greg Chadwick)
[dv] Alter riscv_rf_intg_test to cover more scenarios (Greg Chadwick)
[rtl] Fix logic for generating ECC related alerts (Greg Chadwick)

CHANGE AUTHORIZED: hw/vendor/lowrisc_ibex/rtl/ibex_core.sv

CHANGE AUTHORIZED: hw/vendor/lowrisc_ibex/rtl/ibex_core.sv


Add BUCK filename to Python in languages.yml

BUCK is the filename for the build files of the Facebook Buck build system. BUCK files are valid Python files.
Eg.:
https://github.com/GerritCodeReview/gerrit/blob/master/BUCK
Also add a missing sample for Pants / Bazel BUILD files. They are also valid Python files.

Thanks for this @Dominator008. Do these all look to be BUCK files too?

@afton I skimmed through them and yes they look to be BUCK files.

@afton I skimmed through them and yes they look to be BUCK files.

OK great. Thanks for checking. @Dominator008 - this will be live on GitHub once we release a new Linguist gem which will be in the next week or so.

Thanks!


add test.require() which is equivalent to require with babel hook

We are doing all this work to ensure babel does not leak into any file but the test file, but there is a downside for "helper" files, etc.
The other option would be to hook require and make them provide some sort of include / exclude glob.

Why did you close?

Didn't mean to.

I'm open to it, as it would be very nice to be able to use ES2015 stuff in test helper files too. I think a whitelist would be better so we could still make use of the ES2015 import semantics.

Would be very nice indeed! Maybe use something like .es6.js as extension? This way you don't need to update the whitelist.

Or just default the whitelist to include test/_*, since the underscore prefix is already set aside to denote helpers.
Or just everything in the test directory? Event test/fixtures/**?

@jamestalmage I think defaulting the whitelist to test/_* is a good idea, not so sure about the whole test directory, though. I would definitely not want my fixtures implicitly transpiled.

Thats my feeling as well.

+1, I want this thing too. But why do you need all that stuff around require-from-string?
Why not just add babel hook using require("babel-core/register");?

Because, one goal of this library is not to pollute any of your production code with babel transforms/polyfills (unless you specifically ask it to).

But all my source code is written for babel too. Will there an option to transpile files from my src/ folder in addition to test/?

Because, one goal of this library is not to pollute any of your production code with babel transforms/polyfills (unless you specifically ask it to).

I actually want my source code (non-test) to be transpiled by babel as well..... i.e. I want to be able to import a module that is written in ES6 into my ava test module.  Is there a suggested technique for this today?  Or is this a pending enhancement?

@sparty02 Pending enhancement. Definitely something we want to support, but haven't had the chance yet. See https://github.com/sindresorhus/ava/issues/111. Any thoughts there much appreciated.

@sindresorhus  ahh.... thanks for the reference to #111.  I'll take a look!

Closing this issue as it's already implemented.


Feat/kotlin 1.9.10

Upgrade Kotlin to 1.9.10
Also upgraded to match:

android-tools-gradle to 8.1.0
Ktor client to 2.3.5
Kotlin Coroutines to 1.7.2
Kotlin Serialization to 1.5.1

Going to increase the major version, I don't trust ktor, kotlinx-serialization and kotlinx-coroutines to all be 100% ABI compatible, even though the cognito-idp code is unchanged :eyes:

LGTM


This is currently in progress. The first step is to rid of the internal P: Peer generic.
&mut self for send_reset
While calling this function is the last thing that should be done with
the instance, the intent of the h2 library is not to be used directly by
users, but to be used as an implementation detail by other libraries.
Requiring self on send_reset is pretty annoying when calling the
function from inside a Future implementation. Also, all the other fns
on the type take &mut self.

I have pushed a comment that removes the internal P: Peer generic.

I pushed splitting Respond from server::Stream. This reduces server::Stream and client::Stream to the same API surface.

Ok, I started removing B from public API types.


Sling-5014 - Installer blacklist, to avoid reinstalling older bundles

adding blacklist implementation based on sling_bootstrap.txt

@DominikSuess This has been merged, can you close this PR?

With the upcoming migration to git this repository will be deleted. If the pull request contains data that should be re-submitted I suggest that you save the changes locally and resubmit them after the migration, which will likely happen on Monday the 9th of October.


[VL] Remove VELOX_BUILD_PATH from include directories

What changes were proposed in this pull request?
This is another part of work to reduce dependencies on velox build directory.
How was this patch tested?
CI build.

GHA failure is related:
/__w/incubator-gluten/incubator-gluten/dev/../ep/build-velox/build/velox_ep/velox/dwio/dwrf/common/wrap/dwrf-proto-wrapper.h:34:10: fatal error: velox/dwio/dwrf/proto/dwrf_proto.pb.h: No such file or directory
   34 | #include "velox/dwio/dwrf/proto/dwrf_proto.pb.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [velox/tests/CMakeFiles/velox_plan_conversion_test.dir/build.make:76: velox/tests/CMakeFiles/velox_plan_conversion_test.dir/Substrait2VeloxPlanConversionTest.cc.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:560: velox/tests/CMakeFiles/velox_plan_conversion_test.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 72%] Building CXX object velox/tests/CMakeFiles/velox_operators_test.dir/VeloxColumnarBatchSerializerTest.cc.o
[ 72%] Building CXX object velox/tests/CMakeFiles/velox_operators_test.dir/VeloxColumnarBatchTest.cc.o
[ 73%] Building CXX object velox/tests/CMakeFiles/velox_operators_test.dir/VeloxBatchResizerTest.cc.o
[ 74%] Building CXX object velox/tests/CMakeFiles/velox_operators_test.dir/JsonToProtoConverter.cc.o
[ 74%] Building CXX object velox/tests/CMakeFiles/velox_operators_test.dir/FilePathGenerator.cc.o

GHA failure is related:
/__w/incubator-gluten/incubator-gluten/dev/../ep/build-velox/build/velox_ep/velox/dwio/dwrf/common/wrap/dwrf-proto-wrapper.h:34:10: fatal error: velox/dwio/dwrf/proto/dwrf_proto.pb.h: No such file or directory
   34 | #include "velox/dwio/dwrf/proto/dwrf_proto.pb.h"
      |          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [velox/tests/CMakeFiles/velox_plan_conversion_test.dir/build.make:76: velox/tests/CMakeFiles/velox_plan_conversion_test.dir/Substrait2VeloxPlanConversionTest.cc.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:560: velox/tests/CMakeFiles/velox_plan_conversion_test.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 72%] Building CXX object velox/tests/CMakeFiles/velox_operators_test.dir/VeloxColumnarBatchSerializerTest.cc.o
[ 72%] Building CXX object velox/tests/CMakeFiles/velox_operators_test.dir/VeloxColumnarBatchTest.cc.o
[ 73%] Building CXX object velox/tests/CMakeFiles/velox_operators_test.dir/VeloxBatchResizerTest.cc.o
[ 74%] Building CXX object velox/tests/CMakeFiles/velox_operators_test.dir/JsonToProtoConverter.cc.o
[ 74%] Building CXX object velox/tests/CMakeFiles/velox_operators_test.dir/FilePathGenerator.cc.o

@zhouyuan, yes, just noted the path is required when build test is enabled.


No dynamic PV provisioning with storageclass kubevirt-hostpath-provisioner?

I created a PVC but the PV is not being allocated automatically.
Do I have to manually create every PV like pv-test.sh is doing?
Here is what I tried:
[root@server ~]# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.4 (Ootpa)

[root@server ~]# microshift version
Microshift Version: v0.2-0-g028fbb3

[root@server ~]# kubectl get pvc
NAME                STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS                    AGE
db-noobaa-db-pg-0   Pending                                      kubevirt-hostpath-provisioner   11m

[root@server ~]# kubectl describe pvc
Name:          db-noobaa-db-pg-0
Namespace:     noobaa
StorageClass:  kubevirt-hostpath-provisioner
Status:        Pending
Volume:
Labels:        app=noobaa
               noobaa-db=postgres
Annotations:   volume.beta.kubernetes.io/storage-provisioner: kubevirt.io/hostpath-provisioner
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       noobaa-db-pg-0
Events:
  Type    Reason                Age                 From                         Message
  ----    ------                ----                ----                         -------
  Normal  ExternalProvisioning  92s (x63 over 16m)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "kubevirt.io/hostpath-provisioner" or manually created by system administrator

[root@server ~]# kubectl get pv
No resources found
[root@server ~]# kubectl get sc
NAME                                      PROVISIONER                        RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
kubevirt-hostpath-provisioner (default)   kubevirt.io/hostpath-provisioner   Delete          Immediate           false                  18h

[root@server ~]# kubectl get all -n kubevirt-hostpath-provisioner
NAME                                      READY   STATUS    RESTARTS   AGE
pod/kubevirt-hostpath-provisioner-v7psx   1/1     Running   0          5m51s

NAME                                           DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/kubevirt-hostpath-provisioner   1         1         1       1            1           <none>          18h

[root@server ~]# kubectl logs kubevirt-hostpath-provisioner-v7psx -n kubevirt-hostpath-provisioner
I0701 19:27:32.393613       1 hostpath-provisioner.go:82] initiating kubevirt/hostpath-provisioner on node: server.kube-nfs-kerberos.redhat-et
I0701 19:27:32.395143       1 hostpath-provisioner.go:277] creating provisioner controller with name: kubevirt.io/hostpath-provisioner
I0701 19:27:32.395360       1 controller.go:772] Starting provisioner controller kubevirt.io/hostpath-provisioner_kubevirt-hostpath-provisioner-v7psx_e4930e8e-bc65-48b5-a06e-7ba1c79400a5!
I0701 19:27:32.495474       1 controller.go:821] Started provisioner controller kubevirt.io/hostpath-provisioner_kubevirt-hostpath-provisioner-v7psx_e4930e8e-bc65-48b5-a06e-7ba1c79400a5!
I0701 19:27:32.495548       1 hostpath-provisioner.go:95] isCorrectNodeByBindingMode mode: Immediate

I tried a few manual things and eventually made the PVC bind.
First I created a copy of the storage class with volumeBindingMode: WaitForFirstConsumer and retried the PVC create from scratch, but still no dynamic provisioning.
So I manually created the PV to match:
apiVersion: v1
kind: PersistentVolume
metadata:
  name: db-noobaa-db-pg-0
spec:
  capacity:
    storage: 50Gi
  accessModes:
  - ReadWriteOnce
  hostPath:
    path: "/mnt/share1/db-noobaa-db-pg-0/"

But it didn't bind automatically, so I kubectl edit pv db-noobaa-db-pg-0 and added the claimRef to the Pv spec:
  claimRef:
    name: db-noobaa-db-pg-0
    namespace: noobaa

And then it was bound:
[root@server ~]# kubectl get pvc,pv
NAME                                      STATUS   VOLUME              CAPACITY   ACCESS MODES   STORAGECLASS                                AGE
persistentvolumeclaim/db-noobaa-db-pg-0   Bound    db-noobaa-db-pg-0   50Gi       RWO            kubevirt-hostpath-wait-for-first-consumer   18m

NAME                                 CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                      STORAGECLASS   REASON   AGE
persistentvolume/db-noobaa-db-pg-0   50Gi       RWO            Retain           Bound    noobaa/db-noobaa-db-pg-0                           16m

Oh so I just found in the provisioner logs that it logged when I was using it with WaitForFirstConsumer but still expected me to provide the selected-node annotation on the PVC...
I0701 20:11:18.855836       1 hostpath-provisioner.go:95] isCorrectNodeByBindingMode mode: WaitForFirstConsumer
I0701 20:11:18.855852       1 hostpath-provisioner.go:118] missing volume.kubernetes.io/selected-node annotation, skipping operations for pvc

I followed option 3 mentioned in this readme - https://github.com/kubevirt/hostpath-provisioner#differences so I thought I don't have to set it on the PVC.

this should be fixed in the latest code https://github.com/redhat-et/microshift/pull/114

@guymguym See the latest release
https://github.com/redhat-et/microshift/releases/tag/0.4.7-0.microshift-2021-07-07-002815


system.columns.comment column was introduced only in 18.16.0

https://github.com/ClickHouse/clickhouse-jdbc/blob/5e6f74b89354e60af7d0c9437eef0520ee9ffda1/clickhouse-jdbc/src/main/java/ru/yandex/clickhouse/ClickHouseDatabaseMetadata.java#L804
18.14.18

ClickHouse exception, code: 47, host: ....., port: 8123; Code: 47, e.displayText() =
DB::Exception: Unknown identifier: comment, e.what() = DB::Exception [DB Errorcode=47]

ClickHouse Release 18.16.0
Now you can specify comments for a column in the table description https://github.com/ClickHouse/ClickHouse/pull/3377

Is 18.16 still supported? We can add it into below list for regression.
https://github.com/ClickHouse/clickhouse-jdbc/blob/5e6f74b89354e60af7d0c9437eef0520ee9ffda1/.github/workflows/build.yml#L37
Last time I was told that 19.16(and 20.3?) was no longer supported. I myself think the driver can support wider range. JDK 7 is probably not the case but we can add it back too.

@alexey-milovidov, am I supposed to be use below list for regression? Would be great if there's a similar configuration in ClickHouse server repo.
      {"code": "18.14.19",    "name": "18.14.19 Altinity Stable"},
      {"code": "<IP_ADDRESS>", "name": "<IP_ADDRESS> Altinity Stable"},
      {"code": "<IP_ADDRESS>",  "name": "<IP_ADDRESS> Altinity Stable"},
      {"code": "<IP_ADDRESS>", "name": "<IP_ADDRESS> Altinity Stable"},
      {"code": "<IP_ADDRESS>",   "name": "<IP_ADDRESS> Altinity Stable"},
      {"code": "<IP_ADDRESS>",   "name": "<IP_ADDRESS> Altinity Stable"},
      {"code": "<IP_ADDRESS>",   "name": "<IP_ADDRESS> Altinity Stable"},
      {"code": "<IP_ADDRESS>",   "name": "<IP_ADDRESS> Altinity Stable"},
      {"code": "<IP_ADDRESS>",    "name": "<IP_ADDRESS>"},
      {"code": "<IP_ADDRESS>",   "name": "<IP_ADDRESS>"}

https://github.com/Altinity/ch-stable-versions/blob/c83b2b51e98ea156dd2c31e95c69a41a360f64b8/versions.json#L22-L31

Yes, you can check against these versions.
PS. "Altinity stable" is just a tag for some old frozen release. It does not mean that this release is somehow supported.
Neither it means that release is somehow stable. In fact, old releases contains issues of high severety that were fixed long time ago. Nevertheless, you can find these releases in production for use cases where they work perfectly.

We don't maintain the list of https://github.com/Altinity/ch-stable-versions/blob/c83b2b51e98ea156dd2c31e95c69a41a360f64b8/versions.json#L22-L31 - it contains outdated info and not updated in a timely fashion.

We don't maintain the list of https://github.com/Altinity/ch-stable-versions/blob/c83b2b51e98ea156dd2c31e95c69a41a360f64b8/versions.json#L22-L31 - it contains outdated info and not updated in a timely fashion.

Thanks @alexey-milovidov. Wouldn't it better to maintain a list of all supported versions(mostly in format of major.minor) in a text(yaml/json) file in server repository? That way, all clickhouse clients can use that as inputs for regression test to ensure backward compatibility.

Here is the list: https://github.com/ClickHouse/ClickHouse/pulls?q=is%3Aopen+is%3Apr+label%3Arelease


Added text/html regex to plain serializer

Added uittests
Checklist

[x] Tests have been added (for bug fixes / features)
[ ] Docs have been added / updated (for bug fixes / features)

What kind of change does this PR introduce?
Bug fix
What is the current behavior? What is the new behavior?
Issue link: https://github.com/stoplightio/prism/issues/795
Does this PR introduce a breaking change?
This PR now try to parse json serialiser for text/plain responses as well, since that is how most of the schemas are made by automated tools

So, Instead of merging, I've rebased the master so I can easily squash and merge in the end.

@codeasashu I do not think it's worth to change the serialiser signature just for that reason. Instead, I would probably just reword the message to be "Cannot serialise complex objects as text"

@XVincentX do you still want to keep the error link to https://stoplight.io/prism/errors#NO_COMPLEX_OBJECT_TEXT_PLAIN ?

Yeah let's keep it in that way for now please.

I am unable to run harness tests locally because I somehow deleted the packages/core/dist directory and I do not know how to recreate it (any npm command for that?)

@codeasashu Make sure to remove the package-lock file, we use yarn.

@StefanDywersant Can you take a look at this as well to make sure I did not forget anything?

Do you want me to squash all the changes into single commit?

@codeasashu We'll take care of it.


feat(frontend): Adding routing with router-dom

Related Issue

Info about the related issue

Closes: #[issue number that will be closed through this PR]
Describe the changes you've made
Give a clear description what modifications you have made
Type of change
What sort of change have you made:

[ ] Bug fix (non-breaking change which fixes an issue)
[ ] New feature (non-breaking change which adds functionality)
[x] Enhancement in the existing features
[ ] Other

Mention any unusual behaviour of your code (Write NA if not)
Any unusual behaviour of your code
Checklist:

[x] My code follows the guidelines of this project.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly whereever it was hard to understand.
[x] My changes generate no new warnings.
[x] I have verified/tested my code by running it locally.

Files Added/Modified

[List the files you have added or modified in this pull request.]

Screenshots / Videos / Gif
[Insert screenshots or videos demonstrating the changes you made.]
Additional Info (optional)
Any additional information you want to give

@SantiagoEscalante , start with a lower-case, like adding routing, thanks


ARROW-9534: [Rust] [DataFusion] Add functions to create literal expressions

@jorgecarleitao fyi

Nice! I just needed this in my ballista's PR!
following this SO answer, what do you think of
trait Literal {
    fn lit(&self) -> ScalarValue;
}

impl Literal for i32 {
    fn lit(&self) -> ScalarValue {
        ScalarValue::Int(self.clone())
    }
}
impl Literal for bool {
    fn lit(&self) -> ScalarValue {
        ScalarValue::Boolean(self.clone())
    }
}

fn lit<T: Literal>(n: T) -> ScalarValue {
    n.lit()
}

# usage:
# lit(2);
# lit(true);

this way, the user does not have to import nor remember the value's type; they can just call lit. It seems to be a common pattern in rust.

@jorgecarleitao Oh, that is much nicer. Do you want to PR that and I'll close this one?

I can take it.


Automated cherry pick of #5343: fix: 支持aws rds sync

Cherry pick of #5343 on release/3.1.
#5343: fix: 支持aws rds sync

/lgtm

/approve


tinyscrollbar is not working

tinyscrollbar used to display per element scrollbar is not working

@rclai havent we decided not to use they thinyscrollbar thingy?

I guess we shouldn't, though I didn't know we were using it to being with.


fix(idt): panic in impl fmt::Debug for EntryOptions

When running this test (debug-printing an empty IDT) without the fix, there is a panic:
running 1 test
[src/structures/idt.rs:1725:9] InterruptDescriptorTable::new() = InterruptDescriptorTable {
    divide_error: Entry {
        handler_addr: 0x0,
        options: EntryOptions {
            code_selector: SegmentSelector {
                index: 0,
                rpl: Ring0,
            },
thread 'structures::idt::test::idt_fmt_debug' panicked at src/structures/idt.rs:1006:9:
attempt to subtract with overflow
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test structures::idt::test::idt_fmt_debug ... FAILED

This is because EntryOptions::stack_index subtracts one from the corresponding bits, which are zero for EntryOptions::minimal.
This fix inlines EntryOptions::stack_index into the fmt::Debug implementation and changes the subtraction into a checked subtraction, displaying either None or Some(number) for stack_index in the fmt::Debug implementation.
It might make sense to also change the signature EntryOptions::stack_index to return an Option<u16> itself to enable getting a non-present stack index in user code, but that would be a breaking change.

Thank you for your contribution!

When running this test (debug-printing an empty IDT) without the fix, there is a panic:
running 1 test
[src/structures/idt.rs:1725:9] InterruptDescriptorTable::new() = InterruptDescriptorTable {
    divide_error: Entry {
        handler_addr: 0x0,
        options: EntryOptions {
            code_selector: SegmentSelector {
                index: 0,
                rpl: Ring0,
            },
thread 'structures::idt::test::idt_fmt_debug' panicked at src/structures/idt.rs:1006:9:
attempt to subtract with overflow
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
test structures::idt::test::idt_fmt_debug ... FAILED

This is because EntryOptions::stack_index subtracts one from the corresponding bits, which are zero for EntryOptions::minimal.
This fix inlines EntryOptions::stack_index into the fmt::Debug implementation and changes the subtraction into a checked subtraction, displaying either None or Some(number) for stack_index in the fmt::Debug implementation.

Sounds good to me.

It might make sense to also change the signature EntryOptions::stack_index to return an Option<u16> itself to enable getting a non-present stack index in user code, but that would be a breaking change.

EntryOptions::stack_index is a private method, so this is not a breaking change. It's only used for the Debug impl, so lets go ahead with changing its signature.

EntryOptions::stack_index is a private method, so this is not a breaking change. It's only used for the Debug impl, so lets go ahead with changing its signature.

Ah, that's true! I amended the commit. 👍


use ~ for named args

Fixes https://github.com/facebook/reason/issues/1447

RIP colon. You'll be missed


hyperctl run ignored  --publish=

My ENV
lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.2 LTS
Release:	16.04
Codename:	xenial

uname -a
Linux runv 4.8.0-51-generic #54~16.04.1-Ubuntu SMP Wed Apr 26 16:00:28 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

iptables v1.6.0:

hyperd v0.8.1

hyperd config
cat /etc/hyper/config |grep -v ^$|grep -v ^#
Root=/mnt/hyperdata
Hypervisor=qemu
Kernel=/var/lib/hyper/kernel
Initrd=/var/lib/hyper/hyper-initrd.img
Host=<IP_ADDRESS>:2375
DisableIptables=false
VmFactoryPolicy=
[Log]

After run container as
hyperctl run --memory=2048 --publish=tcp:8811:80 -d nginx
POD id is nginx-0503261496
Time to run a POD is 1485 ms

does not created iptables rules for port map
root@runv:~# iptables -L -vn
Chain INPUT (policy ACCEPT 806 packets, 67939 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  *      hyper0  <IP_ADDRESS>/0            <IP_ADDRESS>/0            ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  hyper0 *       <IP_ADDRESS>/0            <IP_ADDRESS>/0
    0     0 HYPER      all  --  *      hyper0  <IP_ADDRESS>/0            <IP_ADDRESS>/0

Chain OUTPUT (policy ACCEPT 396 packets, 72592 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain HYPER (1 references)
 pkts bytes target     prot opt in     out     source               destination
root@runv:~# iptables -L -vn -t nat
Chain PREROUTING (policy ACCEPT 134 packets, 17336 bytes)
 pkts bytes target     prot opt in     out     source               destination
    3   180 HYPER      all  --  *      *       <IP_ADDRESS>/0            <IP_ADDRESS>/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 102 packets, 15748 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 HYPER      all  --  *      *       <IP_ADDRESS>/0           !<IP_ADDRESS>/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MASQUERADE  all  --  *      !hyper0  <IP_ADDRESS>/24     <IP_ADDRESS>/0

Chain HYPER (2 references)
 pkts bytes target     prot opt in     out     source               destination
root@runv:~#

confirm, it is a known issue of 0.8.x

When the bug fix is planned?

It is listed in the milestone of 0.9.0.

tnx

hello @gnawux  , is there any info about approx 0.9 release date ?

@nickdoikov not yet. However, this is not a feature hard to be implement, just was temporarily removed during the refactor of 0.8.x developing.

@gnawux could you please provide a previous version number where this feature was worked?

@nickdoikov back to 0.7, it should work. But 0.8.x fixed many bugs of previous version, and the port mapping could be done in external systems, such as iptables, kubernetes service, and so on.

@gnawux In new version hyperd there will be native support iptables? Kubernetes is not our case)
I do not want to write the wrapper for iptables myself

@SergeyOvsienko In all the previous, the port mapping is done with iptables

@gnawux I'm interested in new versions? Or in future versions there will be no more support iptables native?

@SergeyOvsienko
In the previous versions, we define port mapping in the spec of each container. While in the new version, the port mapping will become a pod level configuration, and could be update during runtime. The related work item is #382

@gnawux tnx

@gnawux  it is not clear for me now.
do hyperd will support native iptables in future versions?
It is simple question , but it is very important for us.
Why you kill functionality that allow to use hyperd without kubernetes?
Sometimes it is enough to build own schedulers and use common iptables based port forwarding.

@nickdoikov it's just an API/model change, not about the backend implementation. The past is iptables, and the future is iptables too.

@gnawux thanks , so  i can downgrade to 0.7 and build own ecosystem holding in mind that in future i will be capable to upgrade to 0.9 where  port mapping management functionality will be added back?
i mean that  i will  be capable to manage it via REST or cli , not only via kubelet in 0.9 ?

Yes, in the 0.9, both REST/cli and GRPC will be supported.
Note: The life cycle has changed since 0.8.x. (ref: 0.8 release note)

Enhancement: simplify the model and state machine -- one Pod is one VM.
Enhancement: allow add/remove containers to/from a Pod/sandbox.
Enhancement: do not stop sandbox without an explicit stop command even if the last container is stopped (To be compatible to Kubernetes CRI).

manage port publishing ?
or you mean general REST/cli management  ?
My question was about port publishing functionality  and your response not so clear for me, sorry , cloud please clarify  :)
for me is not a problem that mechanisms was changed, i`m worried about functionality (iptables REST/CLI management) .

iptables and cli will all exist in future release. But the interface may be changed.

what you mean when talking about "interface"?
rest endpoints structure? or what ?  not clearly understand you.

For the REST API,

the port mapping spec will move from containers to top level of pod
will have new port mapping add/remove API
will support the port range better

For command line, I think we will try to keep it unchanged.

thanks !
it is clear now .
is there any chance that release 0.9 will be done in 2017 ?

It should be done in 1 or 2 months. And if we could not make a major release, we will release these not-so-big functions in a 0.8.2 release.

cool, many thanks.
Actually we had plans to use hyper in paas platform instead of currently used docker  engine and containers,hyper provide simplicity and excellent resource isolation level.
thats why im very interested in hyper release perspectives.
Thank you one more time.

I am too very interested)

Great, if you met any other issues, feel free to file an issue.

Hello @gnawux
a small update :
We have tested 0.7 and iptables works fine, but this version contain interactive terminal issues.
so we will waiting for 0.8.2 or 0.9.0  hope this will happened soon.


Allow selecting of GPUs

Hashcat tends to be quite greedy on GPU RAM. It would be nice if we could parallelise some of our cracks by allow only certain GPUs to be selected, so we can have, say urgent and heavy jobs on the 3080s and less urgent on the 2080s.
This can be defined using the -d/-D flag to hashcat (where the ids can be found from hashcat -I). Ideally I'd like to have profiles that I can set up as the administrator and let the users choose.
A quick workaround may by to allow custom flags to be applied to the hashcat command option, though this would need careful checking to prevent command injection.

That's a great idea and I like the profile approach.
Thanks for the recommendation, I'll see what I can do.

I pushed this feature to the develop branch - if you have time give it a go and let me know if there are any issues. Profile creation is in the admin panel under "Device Profiles", and you will see a select box in the "Hashcat" session setup step. If a profile is deleted or disabled, the fallback will use all GPUs available (as it does now).
Don't forget to run the db migration for this, instructions at: https://github.com/ctxis/crackerjack/blob/master/docs/installation.md#update-to-latest-version (but checkout the develop instead of the master branch)

Heya, when you clone a fresh version of the develop branch and attemp to log in after creating an admin account, you get the following stack trace:
Traceback (most recent call last):
  File "/tmp/crackerjack/venv/lib/python3.9/site-packages/flask/app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "/tmp/crackerjack/venv/lib/python3.9/site-packages/flask/app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "/tmp/crackerjack/venv/lib/python3.9/site-packages/flask/app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/tmp/crackerjack/venv/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/tmp/crackerjack/venv/lib/python3.9/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/tmp/crackerjack/venv/lib/python3.9/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/tmp/crackerjack/venv/lib/python3.9/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/tmp/crackerjack/venv/lib/python3.9/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/tmp/crackerjack/venv/lib/python3.9/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/tmp/crackerjack/venv/lib/python3.9/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/tmp/crackerjack/app/controllers/home.py", line 26, in index
    errors = healthcheck.run(provider)
  File "/tmp/crackerjack/app/lib/base/healthcheck.py", line 12, in run
    sessions = provider.sessions()
  File "/tmp/crackerjack/app/lib/base/provider.py", line 39, in sessions
    self.device_profiles()
  File "/tmp/crackerjack/app/lib/base/provider.py", line 148, in device_profiles
    return DeviceProfileManager(hashcat.get_detected_devices())
  File "/tmp/crackerjack/app/lib/hashcat/manager.py", line 518, in get_detected_devices
    output = self.shell.execute([self.hashcat_binary, '-I', '--force'], user_id=0, log_to_db=False)
  File "/tmp/crackerjack/app/lib/base/shell.py", line 21, in execute
    output = subprocess.run(command, stdout=subprocess.PIPE).stdout.decode().strip()
  File "/usr/lib/python3.9/subprocess.py", line 505, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.9/subprocess.py", line 951, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib/python3.9/subprocess.py", line 1821, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
PermissionError: [Errno 13] Permission denied: ''

This is probably caused by the healthcheck assuming that the hashcat binary has been set, but it hasn't yet (the admin has to log on first and go to settings).

Thanks for reporting it! I've just pushed a fix to the develop branch.

I've pushed this to master, let me know if there are any issues with it. Thanks.

Yep, it all looks good. Thanks!


optimize sparql query before processing

Done, using jena. We should consider to switch to sesame instead.


Terraform vSphere feature

This feature enables deployment a cluster with Terraform in vSphere environment.
The module uses third-party vSphere provider for Terraform.
For more detailed information regarding this module, please look at README file.

@andreysabitov All tests passed successfully


buffer overflow while parsing if buffer is not nul terminated

Parsing a correct xml from a buffer where the last byte is not '\0' leads to a buffer overflow and an occasional crash depending on the out of buffer random data.

from the comment:
You may optionally pass in the 'nBytes', which is
    	the number of bytes which will be parsed. If not
    	specified, TinyXML-2 will assume 'xml' points to a
    	null terminated string.

So just checking if you passed in the correct arguments? (Re-open if you still have a problem.)


lets prepare publishing env

make sure flatten work
when flatten put small config file in each website/wiki which has domain info
use nodejs webserver to support domains (load the config file with domain info)
fix the dns on/off
https support directly in nodejs (no proxy)
letsencrypt support for nodjs
add https://www.npmjs.com/package/dns2
check memory usage with letsencrypt support nodjs & https & dns

[X] Let's Encrypt
[X] HTTP && HTTPs
[ X] DNS Server


City point icons not showing at node locations

Instead, it is placed at the node position along with the name, while the actual position is between the dot icon and the text.

This has been updated and is now live in our staging site:
https://openhistoricalmap.github.io/mbgl-timeslider/demo/#6.517/39.69301/124.94870/1400,1200-1400

Confirming this is live and done:


 Include clear steps.

opts := append(
  chromedp.DefaultExecAllocatorOptions[:],
  chromedp.WindowSize(CHROME_SCREEN_WIDTH, CHROME_SCREEN_HEIGHT),

  // This is to improve performance by disabling image and plugin
  chromedp.Flag("blink-settings", "imagesEnabled=false"),
  chromedp.ProxyServer("proxy_url"),
)
allocCtx, allocCancel := chromedp.NewExecAllocator(context.Background(), opts...)
defer allocCancel()

browserCtx, browserCancel := chromedp.NewContext(allocCtx)
defer browserCancel()

timeoutCtx, timeoutCancel := context.WithTimeout(browserCtx, 60*time.Second)
defer timeoutCancel()

start := time.Now()

// proxy auth
listenCtx, listenCancel := context.WithCancel(timeoutCtx)
chromedp.ListenTarget(listenCtx, func(ev interface{}) {
  go func() {
    switch ev := ev.(type) {
    case *fetch.EventAuthRequired:
      if ev.AuthChallenge.Source == fetch.AuthChallengeSourceProxy {
        go func() {
          _ = chromedp.Run(timeoutCtx,
            fetch.ContinueWithAuth(ev.RequestID, &fetch.AuthChallengeResponse{
                Response: fetch.AuthChallengeResponseResponseProvideCredentials,
                Username: "proxy_username",
                Password: "proxy_password",
              }),
            fetch.Disable(),
          )
          listenCancel()
        }()
      }
    case *fetch.EventRequestPaused:
      go func() {
        _ = chromedp.Run(timeoutCtx, fetch.ContinueRequest(ev.RequestID))
      }()
    }
  }()
})

err := chromedp.Run(timeoutCtx,
  fetch.Enable().WithHandleAuthRequests(true),
  chromedp.ActionFunc(func(ctx context.Context) error {
    expr := cdp.TimeSinceEpoch(time.Now().Add(180 * 24 * time.Hour))
    return network.SetCookie("name", "1").
      WithExpires(&expr).
      WithDomain("serverurl").
      WithPath("/").
      WithHTTPOnly(false).
      WithSecure(false).
      Do(ctx)
  }),
  chromedp.Navigate("serverurl"),
  chromedp.Click("selector", chromedp.ByQuery),
)

if err != nil {
  log.Println("Error Occurred")
}

What did you expect to see?
I am running a service that when it get hits, it will browse a certain page and click a certain selector. After several months my service's storage is bloated by /tmp/snap-private-tmp/snap.chromium. Is there anyway that I can reduce its disk usage from chromedp setting? Is there anything in my code that is causing this?
What did you see instead?
The storage is bloated with snap.chromium
40G      /tmp/snap-private-tmp/snap.chromium

Let's find out what's in the directory first. Can you share the output of the following command (or something similar)?
$ sudo du /tmp/snap-private-tmp/snap.chromium

I recently rebooted the service, so most of them are gone. Here are what I gathered so far.

It is already around 8GB.

I also notice CPU Usage spiking when I run the code. is it really common to see CPU spiking every time the code runs? Or something in my code is making it spike?
When I tried running it several time (5 to 6 times repeating the same step). I saw spiking around 20% CPU in my local machine (attached image is my processor). I add several steps in my code that does a simple login process on my website (open web, type username, type password, click login, wait for component to show).

Here you go hope this helps...
32      /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/Local Storage/leveldb
36      /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/Local Storage
20      /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/Service Worker/Database
8       /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/Service Worker/ScriptCache/index-dir
148     /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/Service Worker/ScriptCache
8       /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/Service Worker/CacheStorage/f47f09185957d31d0d12165b97c5b1437f6861b2/b110dad2-2b7b-481a-8047-d4bf1c9fad39/index-dir
92      /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/Service Worker/CacheStorage/f47f09185957d31d0d12165b97c5b1437f6861b2/b110dad2-2b7b-481a-8047-d4bf1c9fad39
100     /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/Service Worker/CacheStorage/f47f09185957d31d0d12165b97c5b1437f6861b2
104     /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/Service Worker/CacheStorage
276     /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/Service Worker
296     /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/DawnCache
8       /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/Code Cache/wasm/index-dir
16      /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/Code Cache/wasm
8       /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/Code Cache/js/index-dir
4176    /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/Code Cache/js
4196    /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/Code Cache
20      /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/Session Storage
20      /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/IndexedDB/https_h.online-metrix.net_0.indexeddb.leveldb
28      /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/IndexedDB/https_www.unipin.com_0.indexeddb.leveldb
28      /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/IndexedDB/https_www.midasbuy.com_0.indexeddb.leveldb
80      /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/IndexedDB
32      /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/databases
4       /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/File System/000/t
8       /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/File System/000
20      /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/File System/Origins
32      /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/File System
312     /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/GPUCache
4       /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/blob_storage/833d3fec-8be1-4fc7-b6e3-ccfc38dac8e4
8       /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/blob_storage
64      /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/WebStorage
12      /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/Cache/Cache_Data/index-dir
9656    /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/Cache/Cache_Data
9660    /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default/Cache
15048   /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835/Default
15052   /tmp/snap-private-tmp/snap.chromium/tmp/chromedp-runner779126835

My code simply run login process
chromedp.WaitVisible(email_selector),
chromedp.SendKeys(emailSelector<EMAIL_ADDRESS>chromedp.ByID),
chromedp.Sleep(time.Millisecond),

// Input Unipin Password
chromedp.SendKeys(passwordSelector, "mypass", chromedp.ByID),
chromedp.Sleep(time.Millisecond),

// Click Login Button
chromedp.Click(signInButtonSelector, chromedp.ByID),

After the login process, it should open a new tab. Then, I switch to the new tab using the following code.
var newTabTarget target.ID
for _, tabs := range targets {
  if tabs.OpenerID == currentTargetID {
    newTabTarget = tabs.TargetID
    break
  }
}

if newTabTarget.String() == "" {
  return errors.New("tab not found")
}

newTabCtx, newTabCancel := chromedp.NewContext(timeoutCtx, chromedp.WithTargetID(newTabTarget))
defer newTabCancel()
err = chromedp.Run(newTabCtx,
  // Fetch the name in result
  chromedp.WaitVisible(greeterSelector, chromedp.ByID),
  chromedp.Text(greeterSelector, &name, chromedp.ByID),
)

Please edit the comment to replace the image with the text. Thanks!
And please provide the demo as a full demo that can be run with go run .. And provide a guide describing how to reproduce the issue.

I am unable to share the confidential data in the code (proxy username, proxy password, proxy url, email, password). I will put all confidential data as variable at the top so you can adjust it. So, what it does

Open Page
Input Game ID and Submit
Select Payment Method
Select Product
Submit Purchase (Which will open new tab)
Switch New Tab
Login
Get Price

This is the code, replace .txt to .go before running
sample.txt

The workaround is to create a cron job to remove old directories. For example, the following command finds chromedp-runner* directories that is modified more than 24 hours ago and execute the rm -rf command to remove it.
Note: when find figures out how many 24-hour periods ago the file was  last  modified,  any  fractional part is ignored, so to match -mtime +0, a file has to have been modified at least 1 day ago.
sudo find /tmp/snap-private-tmp/snap.chromium/tmp/ -name 'chromedp-runner*' -type d -mtime +0 -exec rm -rf {} +

I also notice CPU Usage spiking when I run the code. is it really common to see CPU spiking every time the code runs?

Yes. When the code runs, it starts a browser, which usually consumes a remarkable amount of CPU time and memory.

The code runs as a background service and will be triggered by RabbitMQ message. It will send the price it retrieve back to RabbitMQ. Only one instance is running and since it is a private project it has a rather small traffic.
But, from time to time it does exceed timeout (60 seconds) that I set to it (it returns "context deadline exceeded" error) when it gets triggered. I assume that the network from the server is a little unstable when it gets triggered. I am not sure if that might be the cause of the temporary files are not being able to be deleted.
Thanks a lot @ZekeLu  This clears out most of my confusion. At first I thought there is something wrong with the way I code since I am not familiar with how headless browser works 👍 I will try the workaround solution.

The code runs as a background service and will be triggered by RabbitMQ message

Can you share how this is implemented? I want to understand how this could happen and try to check if there is anything the chromedp package can do to address it.

hi sorry for the late reply. I am a little bit busy with some other projects. here is the main service that I run. It is a simple rabbitmq consumer that whenever a certain "command" received I will run chromedp to check pricing.
consumer.txt


feat: implementation of a persistent actor

closes #560

The feature is totally wrong and need more thinking.


Unable to make small map rotation

We need to enable small map view rotations - now, when user rotates for just few degrees (less than 0.09 RAD), the map returns to zero rotation. When it is rotated more, then even the small rotation is possible.
It seems like that the rotation is snapping to 0 degrees with some tolerance.
We need to turn off this snapping.
(It is easily reproducible on standard examples.)

Just configure your ol.View instance with constrainRotation: false.

Works great. Thanks!


Add ability to toggle multithreading on/off from the CLI

For some reason, ROOT::EnableImplicitMT() can be temperamental in ROOT v06.28, e.g., sometimes writing-out to the output file hands indefinitely with MT enabled, so I'd like for this to be enabled/disabled more easily.

Resolved by eddbc54ca6253d21bf67f5474b742c453573e44f


chore: make Partition class getters public

Make getters in Partition class public so that they can be accessed outside com.google.cloud.spanner package.

Why do we need to make this public?

I only need getPartitionToken to be made public for some Google-internal tests to be run. There are some asserts which need to check for the partitionToken, and with default access specifier, it was not accessible outside com.google.cloud.spanner package.
I'll probably have the rest of the getters as default and will make them public in the future if need be.


Fix Dockerfile build

Description
The Docker build used a 2-step installation of requirements
and application.
This was broke by #272.
Contribution Guidelines
Please Read through the Contribution Guidelines.
Fixed Issues
Fixes #300
"BEFORE" and "AFTER" output
To verify that the change works as desired, please include an output of terminal before and after the changes under headings "BEFORE" and "AFTER".
BEFORE
docker build . failed
AFTER
docker build . works
Contribution checklist

[x] I have read the Contributing Guidelines.
[x] The commits refer to an active issue in the repository.
[ ] I have added automated testing to cover this case.

Notes
Changed build file. No code to test.

Codecov Report

Merging #303 into master will not change coverage.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master    #303   +/-   ##
======================================
  Coverage    59.8%   59.8%
======================================
  Files          39      39
  Lines        1928    1928
======================================
  Hits         1153    1153
  Misses        775     775

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fe187bc...ca61cb7. Read the comment docs.

instead of the setuptools custom command, why not just run pip install ., which will automatically discover packages in setup.cfg?
why are we using --editable with pip install?

In order to use docker build cache properly, we need to separate dependencies installation apart from kube-hunter installation. Using pip install . installs both so it can't be split into two steps.
--editable is meant mostly for developing/debugging purposes. The install make target shouldn't use it.


[SPARK-24881][SQL] New Avro option - compression

What changes were proposed in this pull request?
In the PR, I added new option for Avro datasource - compression. The option allows to specify compression codec for saved Avro files. This option is similar to compression option in another datasources like JSON and CSV.
Also I added the SQL configs spark.sql.avro.compression.codec and spark.sql.avro.deflate.level. I put the configs into SQLConf. If the compression option is not specified by an user, the first SQL config is taken into account.
How was this patch tested?
I added new test which read meta info from written avro files and checks avro.codec property.

Test build #93404 has started for PR 21837 at commit c5802df.

Can one of the admins verify this patch?

Can one of the admins verify this patch?

Test build #93405 has started for PR 21837 at commit f8b580b.

Test build #93404 has finished for PR 21837 at commit c5802df.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93404/
Test FAILed.

jenkins, retest this, please

Test build #93408 has started for PR 21837 at commit f8b580b.

Test build #93405 has finished for PR 21837 at commit f8b580b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93405/
Test PASSed.

Merged build finished. Test PASSed.

Test build #93408 has finished for PR 21837 at commit f8b580b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93408/
Test PASSed.

Test build #93461 has started for PR 21837 at commit d21d3e9.

is there any or easy way to check the metadata for compression level?

@HyukjinKwon I am not sure the level exists in the metadata. At least avro-tools doesn't show it.
java -jar ~/avro-tools-1.7.7.jar getmeta ./part-00000-780d0d02-ec8c-407e-8427-1790b81a3726-c000.avro
avro.schema	{"type":"record","name":"topLevelRecord","namespace":".topLevelRecord","fields": ...}
avro.codec	deflate

Test build #93461 has finished for PR 21837 at commit d21d3e9.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93461/
Test PASSed.

Test build #93632 has started for PR 21837 at commit b315f37.

Test build #93634 has started for PR 21837 at commit 6915f34.

Test build #93636 has started for PR 21837 at commit 5561582.

Test build #93637 has started for PR 21837 at commit ebaf327.

Test build #93632 has finished for PR 21837 at commit b315f37.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93632/
Test PASSed.

Merged build finished. Test PASSed.

Test build #93634 has finished for PR 21837 at commit 6915f34.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93634/
Test PASSed.

Merged build finished. Test PASSed.

Test build #93637 has finished for PR 21837 at commit ebaf327.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93637/
Test PASSed.

Test build #93658 has started for PR 21837 at commit 5f83902.

Test build #93658 has finished for PR 21837 at commit 5f83902.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test FAILed.

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93658/
Test FAILed.

jenkins, retest this, please

Test build #93659 has started for PR 21837 at commit 5f83902.

Test build #93659 has finished for PR 21837 at commit 5f83902.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

Merged build finished. Test PASSed.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/93659/
Test PASSed.

Merged to master.


OData 2.0 $type parameter

In OData 2.0 it is possible to retrieve data in either JSON or XML format by specifying the $format parameter. It would be nice to have this parameter available via the module as well.
Right now as a workaround I use:
.transformUrl(function(url){
        return url+'&$format=json';
      })

resource.odata().format('json').query(); will return what you desire.
Guess we need to get that in the documentation.

Thanks! This indeed solves the issue!


chore: move e2e tests into component source dirs

I'm naming e2e test files *.e2e.spec.ts to differentiate them from unit tests, and adding the e2e tag to e2e blaze targets to differentiate those.

Thanks @devversion for helping with the pile of CI failures!

Awesome! the macro for the e2e tests really reduces the duplication :+1:


API improvement: Specify digest of artifact in certificate

Description

Consider the following attack:

Client generates keypair K
Client fetches a Fulcio certificate
Client uploads signature over artifact and Fulcio certificate to Rekor log
A malicious process (MP) steals K and certificate
MP uploads a signature over its own artifact and Fulcio certificate to Rekor log

Currently the only defense against this attack is the limited lifetime of the certificate and the ephemerality of the keypair.
If the Fulcio certificate were issued not just for an identity but also for an artifact, then this attack would not be possible. On verification, a client would detect that the artifact in the certificate does not match the signed artifact.
To implement this, we need to:

Add an OID for the digest of an artifact
Add an optional API field for the digest
Include the digest in the certificate

Cosign can check for the matching digest. Possibly Rekor could do before uploading to the log, though I'm not sure if we're currently validating the fields of the certificate.
cc @asraa @bobcallaway

Add an optional API field for the digest

Let's add some strict verification here for a digest here. I would be a little worried about user-defined OID extensions (does Private CA fuzz? hahaha)

Possibly Rekor could do before uploading to the log, though I'm not sure if we're currently validating the fields of the certificate.

I think this would be suitable for a monitor! I'd assume if an attacker were going at these lengths, they wouldn't be using cosign and be manually uploading to rekor :)

cc @laurentsimon

This also protects against

Entry malleability
The need to distribute separate sig

However, this relies on trusting the CA (which we already do), but the delegation of the trust is different. The CA is trusted to sign code-signing certs, not to attest data or content.
Atlernatives:

Add the signature as an optional extension. This would also prevent against any of the risks/threats. It's kind of a bind between the public key and the signature.

This also protects against

Entry malleability
The need to distribute separate sig

However, this relies on trusting the CA (which we already do), but the delegation of the trust is different. The CA is trusted to sign code-signing certs, not to attest data or content.

Not sure this is fully true. The leaf cert is only valid for code signing, so verification should fail unless clients disregard the v3 extension. That's not directly related to the proposal in the issue. What I mean in that in the current implementation, clients may also disregard the extension. So I don't think this changes the trust model; it just removes on hop in the chain.
An alternative would be to have 2 root certs: one that delivers leaf certs, and one that signs (with the code-signing extension). My intuition is that 99% f OIDC use cases have a single hop chain, so optimizing for it seems legitimate.
I agree this is not a general solution to malleability, and you may still want to address it in the general case.
I don't know what the definition of code-signing is, but provenance data is not just "code". It's typically related to code one way or another, not it's not code per se: scanning results, SBOMs, scorecard results, etc. I think this is fine, but just curious if you have more info/thoughts on this :-)
Happy to chat more

IMO in any scenario where the ephemeral key leaks, the OIDC token could leak too so this doesn't really solve the issue. "My laptop is owned" isn't really a threat model I think we're ever going to be able to handle.
A better solution here is generate the keys inside a touch-gated enclave or similar and bind them to the OIDC token (DPoP). This doesn't solve the problem, of course, but nothing will.

Is this something there is still interest in? I don't think we'll pursue this otherwise and I'll close it out.

Going to close this as I don't think we'll be implementing this. Feel free to reopen if this comes up again!


Fetch market price using web2

Depends on #53
Faster alternative to #54
Use Alchemy or another SDK to fetch price data for WSTETH, RETH, and ODG

Closing as won't do -- part of TVL project


Preserve bets across restarts

This should enable us to simply restart the bot on getting a RECONNECT command.

Sweet :smile:
Promise style nit!

Should be done :smile:

This should be it now... Promises are difficult.


notebook of figures from the original scVI paper

Let's add a Jupyter notebook to the doc/examples directory that shows how to generate the figures (and one table) from the original scVI paper. It isn't necessary to reconstruct the parts of the figures and the table about competing methods that we compare to---we just need the parts about scVI.

[ ] Figure 1c. runtime; BRAIN_LARGE
[ ] Table 1. log likelihood; BRAIN_LARGE
[ ] Figure 2. imputation; CORTEX
[ ] Figure 3. interpreting latent space; tsne plot and heatmap; CORTEX, HEMATO, random zinb
[ ] Figure 4. batch effect removal; tsne plot; RETINA
[ ] Figure 5. differential expression; plots; PBMC
[ ] Figure 6. plots; PBMC
[ ] Figure 7. posterior plots; CORTEX
[ ] Figure 8. silhouette and batch mixing entropy; PBMC, CORTEX, CBMC, RETINA
[ ] Figure 9. imputation; CORTEX
[ ] Figure 10. imputation; CORTEX
[ ] Figure 11. imputation; CORTEX and BRAIN_LARGE
[ ] Figure 12. DE; plots; PBMC
[ ] Figure 13. DE; plots; BRAIN_SMALL
[ ] Figure 14. imputation; BRAIN_LARGE, CORTEX
[ ] Figure 15. clustering; random and HEMATO
[ ] Figure 16. tsne; HEMATO

Do you remember how many epochs you used for the runtime figure (figure 1)? I only found the runtime code for other softwares on reproducibility notebook > <

^ @romain-lopez

Hi Yining,
For this updated figure, let us use max_epochs = 125 but with early stopping (it should stop way before 125 epochs).
Romain


There is a typo issue on refernce page

https://react.dev/reference/react-dom/components/common#reference here you can find

Fixed by https://github.com/reactjs/react.dev/pull/6167


Improve clarity of (s, mu) internals

As suggested by @surhudm and pointed out by @johannesulf - do not use variable name mu as this is misleading for future developers inspecting the code internals.

This is resolved by e8ac9772b0e2a09205e876baba3407b2841910d3


Scroll window while dragging to top

In examples page http://angular-dragdrop.github.io/angular-dragdrop/examples/ there is possibility to scroll bottom while dragging element near bottom edge of window, but you can't scroll in the same way to top while dragging near top edge.
Tested o chrome 49.

On firefox 45.0.1 it doesn't work in both directions.

It will work in chrome, but it does NOT work in firefox.
here is an example https://plnkr.co/edit/RGE82IoYzEOCB73W2aqu?p=preview

do you have some solution with the last version of this lib ?


Fix apriltag coordinate system notes

can you add camera-coord.png? its in the images folder but never actaully on that page

Lol I do


[Not for land] Added changes for GPT-2 perf

Stack from ghstack (oldest at bottom):

-> #533
#532

Credit: @felipemello1 for most of the work here (especially around chunked cross entropy)
Running on 4xH100s:
Without these changes (torch.compile), the max local batch size is 5:
[rank0]:2024-08-19 11:10:26,196 - root - INFO - Training starts at step 1, with local batch size 5, global batch size 20, sequence length 8192, total steps 100 (warmup 200)
[rank0]:/data/users/andgu/pytorch/torch/_inductor/lowering.py:1673: UserWarning: Torchinductor does not support code generation for complex operators. Performance may be worse than eager.
[rank0]:  warnings.warn(
[rank0]:2024-08-19 11:10:33,811 - root - INFO - step:  1  loss: 12.2365  memory: 81.67GiB(85.93%)  wps: 5,380  mfu: 1.09%
[rank0]:2024-08-19 11:10:33,811 - root - INFO - Synchronizing and adjusting timeout for all ProcessGroups to 0:01:40
[rank0]:2024-08-19 11:10:37,109 - root - INFO - step: 10  loss: 12.1951  memory: 81.67GiB(85.93%)  wps: 111,770  mfu: 22.68%
[rank0]:2024-08-19 11:10:40,777 - root - INFO - step: 20  loss: 11.9455  memory: 81.67GiB(85.93%)  wps: 111,714  mfu: 22.67%
[rank0]:2024-08-19 11:10:44,428 - root - INFO - step: 30  loss: 11.0407  memory: 81.67GiB(85.93%)  wps: 112,194  mfu: 22.76%
[rank0]:2024-08-19 11:10:48,083 - root - INFO - step: 40  loss:  9.9520  memory: 81.67GiB(85.93%)  wps: 112,109  mfu: 22.75%
[rank0]:2024-08-19 11:10:51,734 - root - INFO - step: 50  loss:  9.3392  memory: 81.67GiB(85.93%)  wps: 112,218  mfu: 22.77%
[rank0]:2024-08-19 11:10:55,386 - root - INFO - step: 60  loss:  8.7255  memory: 81.67GiB(85.93%)  wps: 112,198  mfu: 22.77%
[rank0]:2024-08-19 11:10:59,037 - root - INFO - step: 70  loss:  8.1659  memory: 81.67GiB(85.93%)  wps: 112,234  mfu: 22.77%
[rank0]:2024-08-19 11:11:02,701 - root - INFO - step: 80  loss:  7.8037  memory: 81.67GiB(85.93%)  wps: 111,802  mfu: 22.68%
[rank0]:2024-08-19 11:11:06,361 - root - INFO - step: 90  loss:  7.5327  memory: 81.67GiB(85.93%)  wps: 111,937  mfu: 22.71%
[rank0]:2024-08-19 11:11:10,026 - root - INFO - step: 100  loss:  7.3730  memory: 81.67GiB(85.93%)  wps: 111,803  mfu: 22.69%

Without these changes (no torch.compile), local batch size 5:
[rank0]:2024-08-19 14:24:32,150 - root - INFO - Training starts at step 1, with local batch size 5, global batch size 20, sequence length 8192, total steps 100 (warmup 200)
[rank0]:2024-08-19 14:24:38,558 - root - INFO - step:  1  loss: 12.2581  memory: 86.47GiB(90.99%)  wps: 6,393  mfu: 1.30%
[rank0]:2024-08-19 14:24:38,558 - root - INFO - Synchronizing and adjusting timeout for all ProcessGroups to 0:01:40
[rank0]:2024-08-19 14:24:42,308 - root - INFO - step: 10  loss: 12.2099  memory: 86.48GiB(90.99%)  wps: 98,305  mfu: 19.95%
[rank0]:2024-08-19 14:24:46,482 - root - INFO - step: 20  loss: 11.9421  memory: 86.48GiB(90.99%)  wps: 98,230  mfu: 19.93%
[rank0]:2024-08-19 14:24:50,648 - root - INFO - step: 30  loss: 11.0090  memory: 86.48GiB(90.99%)  wps: 98,435  mfu: 19.97%
[rank0]:2024-08-19 14:24:54,788 - root - INFO - step: 40  loss:  9.9780  memory: 86.48GiB(90.99%)  wps: 99,064  mfu: 20.10%
[rank0]:2024-08-19 14:24:58,936 - root - INFO - step: 50  loss:  9.3572  memory: 86.48GiB(90.99%)  wps: 98,813  mfu: 20.05%
[rank0]:2024-08-19 14:25:03,181 - root - INFO - step: 60  loss:  8.7479  memory: 86.48GiB(90.99%)  wps: 96,567  mfu: 19.59%
[rank0]:2024-08-19 14:25:07,339 - root - INFO - step: 70  loss:  8.1769  memory: 86.48GiB(90.99%)  wps: 98,604  mfu: 20.01%
[rank0]:2024-08-19 14:25:11,497 - root - INFO - step: 80  loss:  7.8070  memory: 86.48GiB(90.99%)  wps: 98,579  mfu: 20.00%
[rank0]:2024-08-19 14:25:15,649 - root - INFO - step: 90  loss:  7.5329  memory: 86.48GiB(90.99%)  wps: 98,743  mfu: 20.04%
[rank0]:2024-08-19 14:25:19,798 - root - INFO - step: 100  loss:  7.3700  memory: 86.48GiB(90.99%)  wps: 98,818  mfu: 20.05%

With these changes, we can use local batch size 16:
[rank0]:2024-08-19 11:16:09,534 - root - INFO - Training starts at step 1, with local batch size 16, global batch size 64, sequence length 8192, total steps 100 (warmup 200)
[rank0]:/data/users/andgu/pytorch/torch/_inductor/lowering.py:1673: UserWarning: Torchinductor does not support code generation for complex operators. Performance may be worse than eager.
[rank0]:  warnings.warn(
[rank0]:2024-08-19 11:16:15,523 - root - INFO - step:  1  loss: 12.2386  memory: 72.29GiB(76.06%)  wps: 21,887  mfu: 4.44%
[rank0]:2024-08-19 11:16:15,523 - root - INFO - Synchronizing and adjusting timeout for all ProcessGroups to 0:01:40
[rank0]:2024-08-19 11:16:22,538 - root - INFO - step: 10  loss: 12.1966  memory: 72.30GiB(76.07%)  wps: 168,174  mfu: 34.12%
[rank0]:2024-08-19 11:16:30,332 - root - INFO - step: 20  loss: 11.9229  memory: 72.30GiB(76.07%)  wps: 168,196  mfu: 34.13%
[rank0]:2024-08-19 11:16:38,129 - root - INFO - step: 30  loss: 10.9399  memory: 72.30GiB(76.07%)  wps: 168,144  mfu: 34.12%
[rank0]:2024-08-19 11:16:45,937 - root - INFO - step: 40  loss:  9.8742  memory: 72.30GiB(76.07%)  wps: 167,898  mfu: 34.07%
[rank0]:2024-08-19 11:16:53,734 - root - INFO - step: 50  loss:  9.2517  memory: 72.30GiB(76.07%)  wps: 168,130  mfu: 34.11%
[rank0]:2024-08-19 11:17:01,518 - root - INFO - step: 60  loss:  8.6441  memory: 72.30GiB(76.07%)  wps: 168,435  mfu: 34.18%
[rank0]:2024-08-19 11:17:09,279 - root - INFO - step: 70  loss:  8.0827  memory: 72.30GiB(76.07%)  wps: 168,927  mfu: 34.28%
[rank0]:2024-08-19 11:17:17,047 - root - INFO - step: 80  loss:  7.7330  memory: 72.30GiB(76.07%)  wps: 168,772  mfu: 34.24%
[rank0]:2024-08-19 11:17:25,139 - root - INFO - step: 90  loss:  7.4835  memory: 72.30GiB(76.07%)  wps: 162,008  mfu: 32.87%
[rank0]:2024-08-19 11:17:32,944 - root - INFO - step: 100  loss:  7.3274  memory: 72.30GiB(76.07%)  wps: 167,963  mfu: 34.08%

22.7% MFU -> 34.1% MFU

Great results! Consider using .split(8192//32, dim=1) instead of .chunk(16).
There was a huge difference in reserved memory in my experiments (no change in active memory though)

FYI, compiling loss + model together should yield much better results than the current torch.compile.
instead of doing: torch.compile(model)
do something like:
@torch.compile()
def loss_step(input,label):
output = self.model(input)
loss = calculate_loss(output, label)
loss.backward()
What we found is that using torch.compile on the cross entropy loss alone has great memory benefits (but not better than chunked): https://fb.workplace.com/groups/257735836456307/permalink/708422718054281/
And doing model + loss together almost doubles toks/second: https://github.com/pytorch/torchtune/issues/1228#issuecomment-2277151232

If I try to compile both the output linear and cross entropy loss together instead of just compiling the cross entropy loss, I get OOMs at the same batch size.

My uneducated guess is that the optimizations they made for CrossEntropyLoss accounts only for the loss being compiled on its own. Details of their implementation here: https://fb.workplace.com/groups/257735836456307/permalink/708422718054281/

With these changes:
[rank0]:2024-08-21 08:44:32,865 - root - INFO - Model llama3 8B size: 8,030,261,248 total parameters
[rank0]:2024-08-21 08:44:32,897 - root - INFO - Compiling each TransformerBlock with torch.compile
[rank0]:2024-08-21 08:44:32,953 - root - INFO - Applied FSDP to the model
[rank0]:NCCL version 2.21.5+cuda12.0
[rank0]:2024-08-21 08:44:45,742 - root - INFO - GPU memory usage for model: 3.99GiB(4.20%)
[rank0]:2024-08-21 08:44:45,743 - root - INFO - Training starts at step 1, with local batch size 1, global batch size 8, sequence length 8192, total steps 1000 (warmup 200)
[rank0]:/data/users/andgu/pytorch/torch/_inductor/lowering.py:1673: UserWarning: Torchinductor does not support code generation for complex operators. Performance may be worse than eager.
[rank0]:  warnings.warn(
[rank0]:2024-08-21 08:46:07,756 - root - INFO - step:  1  loss: 12.2044  memory: 71.90GiB(75.65%)  wps: 100  mfu: 0.58%
[rank0]:2024-08-21 08:46:07,756 - root - INFO - Synchronizing and adjusting timeout for all ProcessGroups to 0:01:40
[rank0]:2024-08-21 08:46:19,266 - root - INFO - step: 10  loss: 10.8650  memory: 82.42GiB(86.72%)  wps: 6,405  mfu: 37.51%
[rank0]:2024-08-21 08:46:32,000 - root - INFO - step: 20  loss:  9.1536  memory: 82.42GiB(86.72%)  wps: 6,434  mfu: 37.68%
[rank0]:2024-08-21 08:46:44,771 - root - INFO - step: 30  loss:  8.1057  memory: 82.42GiB(86.72%)  wps: 6,416  mfu: 37.57%

Baseline:
[rank0]:2024-08-21 08:47:49,505 - root - INFO - Model llama3 8B size: 8,030,261,248 total parameters
[rank0]:2024-08-21 08:47:49,829 - root - INFO - Compiling each TransformerBlock with torch.compile
[rank0]:2024-08-21 08:47:49,892 - root - INFO - Applied FSDP to the model
[rank0]:NCCL version 2.21.5+cuda12.0
[rank0]:2024-08-21 08:48:01,630 - root - INFO - GPU memory usage for model: 3.78GiB(3.98%)
[rank0]:2024-08-21 08:48:01,631 - root - INFO - Training starts at step 1, with local batch size 1, global batch size 8, sequence length 8192, total steps 1000 (warmup 200)
[rank0]:/data/users/andgu/pytorch/torch/_inductor/lowering.py:1673: UserWarning: Torchinductor does not support code generation for complex operators. Performance may be worse than eager.
[rank0]:  warnings.warn(
[rank0]:2024-08-21 08:48:56,359 - root - INFO - step:  1  loss: 12.2556  memory: 67.73GiB(71.26%)  wps: 150  mfu: 0.88%
[rank0]:2024-08-21 08:48:56,359 - root - INFO - Synchronizing and adjusting timeout for all ProcessGroups to 0:01:40
[rank0]:2024-08-21 08:49:08,017 - root - INFO - step: 10  loss: 10.8891  memory: 71.64GiB(75.38%)  wps: 6,324  mfu: 37.03%
[rank0]:2024-08-21 08:49:20,989 - root - INFO - step: 20  loss:  9.0440  memory: 71.64GiB(75.38%)  wps: 6,316  mfu: 36.99%
[rank0]:2024-08-21 08:49:33,984 - root - INFO - step: 30  loss:  8.0371  memory: 71.64GiB(75.38%)  wps: 6,305  mfu: 36.92%
[rank0]:2024-08-21 08:49:46,985 - root - INFO - step: 40  loss:  7.4550  memory: 71.64GiB(75.38%)  wps: 6,302  mfu: 36.90%
[rank0]:2024-08-21 08:50:00,009 - root - INFO - step: 50  loss:  7.2264  memory: 71.64GiB(75.38%)  wps: 6,290  mfu: 36.84%


Add release note: sparse tensor support in tf.data

Thank you for catching this in the release notes.  Too much time has likely passed although it could still be merged.  If you go back and sign the CLA please tag me and I can reopen and try to get this merged.
I think you get an email but if not this link should make it easy and the official links are above in the comments from the bot.
https://cla.developers.google.com/clas