Add auto deploy

@sindresorhus Same deal as before, and this is pretty much a wholesale copy from Refined GitHub except for the extension's appID.
From the last time...

To get this working we need to set some environment variables for Travis CI. You can do this by adding encrypted credentials to the .travis.yml file, or through the Travis CI website. I documented it a bit here https://github.com/paulmolluzzo/test-deploy-chrome

Once that is done, we actually need to have this in master because the bash script is going to check that we're on that branch when it runs the script. You can test this outside of master by removing the checks in ./deploy.sh, but you'll want to put them back in so we aren't deploying non-master branches or PRs, which are likely WIP or something.

How about we try out @DrewML's https://github.com/DrewML/chrome-webstore-upload?

That works for me, except I think the CLI version might be more appropriate, no? https://github.com/DrewML/chrome-webstore-upload-cli
That said we'll still need the environment variables on Travis CI to make this work, assuming we want the same workflow of "deploy when a tagged version is pushed and passes".

That would be awesome :) Gives me more opportunity to find places where my code breaks 😎
Let me know if you have any problems getting it setup. I was going to submit a PR to refined-github soon to drop grunt and swap this in.
If we use the same env vars as we do in the other project, you'll just need to do:
webstore upload --file extension.zip --client-id hlepfoohegkhhmjieoechaddaejaokhf --auto-publish

You could also move --client-id to an unsecured env var in .travis.yml and name it CLIENT_ID.

@DrewML I'm already hitting some snags and was about to move onto the non-CLI version and put this into a separate node script (i.e. "deploy": "node deploy-script.js". I don't know if it's better to make a ticket there or put it here, so I'll drop it here and I can make tickets as you like on the appropriate repo later:
The first issue is that we have to have these as env vars for security, so the deploy script I'm looking at is:
"deploy": "webstore upload --file extension.zip --extension-id hlepfoohegkhhmjieoechaddaejaokhf  --client-id $CLIENT_ID --client-secret $CLIENT_SECRET --refresh-token $REFRESH_TOKEN",

The docs suggest that I can omit the flags if I have environment variables, but that doesn't actually work. If I try your suggestion above I get an error: Option "extensionId" is required. (Side note, the client-id is for the API and should be secure, the extension-id is the public ID of the extension itself.)
Originally I followed the documentation, but after trying a few combinations the only working scenario was if I explicitly wrote the variables into the script command. The env vars are printing exactly like they're written in the command (it echoes with the $CLIENT_ID, etc), and swapping in process.env.CLIENT_ID, etc. doesn't fix that either.
Here's where this stands, as far as I can tell: We can try to figure a way to get the env vars into the command (not really ideal), or we can try the non-CLI version and a dedicated deploy script. Of course, if @DrewML wants/can modify the npm package to work as described in the documentation, we can just go with his suggestion. 😄
I don't have access to the secrets for this repo, so I am testing this with my test deploy chrome repo and the secrets I have there. I doubt this is specific to my repo, and like I said, I got this working by explicitly entering the secrets like this (note I have to put them in ' as well):
"deploy": "webstore upload --file extension.zip  --extension-id 'gppbnpldokpcchfpckfdoiaeboinfldm' --client-id 'secret' --client-secret 'secret2' --refresh-token 'secret3'",

Originally I followed the documentation, but after trying a few combinations the only working scenario was if I explicitly wrote the variables into the script command. The env vars are printing exactly like they're written in the command (it echoes with the $CLIENT_ID, etc), and swapping in process.env.CLIENT_ID, etc. doesn't fix that either.

That's interesting - it's working fine with the environment variables for OctoPreview. That extension is running a slightly older version, though, so I installed the latest from npm and ran it locally:
┌─[alevine] [~/test-chrome-extension]
└─▪ webstore upload --extension-id afdllpgoebbbnpfjnhnepldelicbncjd --file extension.zip --auto-publish
Publish Status: OK
Details: OK

I'll take a look at the setup in your repo

Here are the relevant bits of code where those options are used.
Parsing of options/favoring of env vars    and tests
Usage of options

OK, I think I set up the environment variables wrong locally earlier and only set it for the current shell. When I set this up with Travis CI it worked on the first shot.
I'm going to send up a PR for refined-twitter soon.

This can be closed in favor of #35. Even if that's not merged yet it'll be the one we go with I'm sure.


How to keep sol::user_type type as meta when I use sol_lua_push to generate custom userdata?

Thanks for your help!
My code:
class Test {
public:
    void Print() const {
        std::cout << _i << std::endl;
    };
private:
    int _i = 333;
};
template<typename T, typename = typename std::enable_if<std::is_base_of<Test, T>::value>::type>
int sol_lua_push(lua_State* L, const T* obj) {
    // I want to control the generation of userdata myself to avoid duplication of userdata when passing pointers with c++.
    // And I want to do lua_ref on the generated userdata to keep the reference in my system.
    // So, I want to use the c api to create the userdata.
    if (check_userdata_exist(obj)) {
        push_the_exist_userdata(obj);
    }
    else {
        *static_cast<const T**>(lua_newuserdata(L, sizeof(const T*))) = obj;
        sol::userdata ud = sol::userdata(L, -1);
        // But how to use user_type<Test> as the meta?
    }
    return 1;
}

int main() {
    sol::state lua = {};
    lua.open_libraries();
    auto ut = lua.new_usertype<Test>("Test", sol::factories([]() {return new Test();}),
        "Print", &Test::Print
        );
    lua.script("local test = Test.new();test:Print();");
    // error:attempt to index a userdata value (local 'test')
    return 0;
}

I thought of a way to use sol.ClassName* to get the meta table, but will this work universally?

I thought of a way to use sol.ClassName* to get the meta table, but will this work universally?

No. That's a detail and I reserve the right to break it. (And I am going to break it, soon, because I am going to introduce optimizations to the name to save on speed and also give people the ability to change what it returns; it may not even be a string anymore...)
If you want to take control of a userdata at this level and change how the metatable works, you need to make your own metatable. That means giving up a lot of sol2's usertype niceness to craft your own metatables.
I would recommend just pushing the pointer into sol2 the normal way after doing it, since it will have the exact same memory layout you are doing here anyways: https://sol2.readthedocs.io/en/latest/api/usertype_memory.html
For this, you need to pull the latest develop branch (because there was a fix I needed to do to make this work properly), then write this:
class Test {
public:
	void Print() const {
		std::cout << _i << std::endl;
	};

private:
	int _i = 333;
};

bool check_userdata_exist(const Test*) {
	return false; // not real implementation, just for test
}

int push_the_exist_userdata(const Test*) {
	return 1;  // not real implementation, just for test
}

template <typename T, typename = typename std::enable_if<std::is_base_of<Test, T>::value>::type>
int sol_lua_push(lua_State* L, const T* obj) {
	// I want to control the generation of userdata myself to avoid duplication of userdata when passing pointers with c++.
	// And I want to do lua_ref on the generated userdata to keep the reference in my system.
	// So, I want to use the c api to create the userdata.
	if (check_userdata_exist(obj)) {
		 return push_the_exist_userdata(obj);
	}

	// push new userdata into sol2 and hook it up
	// to the normal metatable
	sol::stack::push_userdata(L, obj);
	return 1;
}

int main() {
	sol::state lua = {};
	lua.open_libraries();
	auto ut = lua.new_usertype<Test>("Test", sol::factories([]() { return new Test(); }), "Print", &Test::Print);
	lua.script("local test = Test.new();test:Print();");
	return 0;
}

If you want to take control of a userdata at this level and change how the metatable works, you need to make your own metatable.

No,I don't want to use my metatable,i just want to use myself's userdata and set metatable by sol2's user_type.


Add link id on links and automatic Grafana dashboard

Proposed fix to issue #4

Currently the import of the dashboard is manual... but this can be further automated:

In grafana: New + Import + Paste JSON
*The metrics names need to be processed, so in case of gnmic, these processors need to be used:

processors:
  trim-prefixes:
    event-strings:
      value-names:
        - "^/state/.*"
      transforms:
        - trim-prefix:
            apply-on: "name"
            prefix: "/state/"
  oper-state-to-int:
    event-strings:
      value-names:
        - ".*"
      transforms:
        - replace:
            apply-on: "value"
            old: "up"
            new: "1"
        - replace:
            apply-on: "value"
            old: "down"
            new: "0"
  rename-port-sros:
    event-strings:
      tag-names:
        - "^port_port-id"
      transforms:
        - replace:
            apply-on: "name"
            old: "port_port-id"
            new: "interface_name"
  rename-srl-interface:
    event-strings:
      tag-names:
        - "^interface_name"
      transforms:
        - replace:
            apply-on: "value"
            old: "ethernet-(\\d+)/(\\d+)"
            new: "e$1-$2"

SROS links need to be posprocessed as well but this work for SRL in the meantime


WinAppDriver not finding UWP app by ID even though the app is installed

Proof that my UWP app is installed (Visual Studio installs it when I run it in Debug configuration):
PS > Get-AppxPackage -AllUsers | ?{ $_.Name.Contains("52a95998-8cfa-43d7-96c9-06fc6d8ad1c5") }

Name                   : 52a95998-8cfa-43d7-96c9-06fc6d8ad1c5
Publisher              : CN=***
Architecture           : X64
ResourceId             :
Version                : <IP_ADDRESS>
PackageFullName        : 52a95998-8cfa-43d7-96c9-06fc6d8ad1c5_<IP_ADDRESS>_x64__x38zh4nq6sjk4
InstallLocation        : C:\***\src\XamlUwpApp\bin\x64\Debug\AppX
IsFramework            : False
PackageFamilyName      : 52a95998-8cfa-43d7-96c9-06fc6d8ad1c5_x38zh4nq6sjk4
PublisherId            : x38zh4nq6sjk4
PackageUserInformation : {*** [***\***]: Installed}
IsResourcePackage      : False
IsBundle               : False
IsDevelopmentMode      : True
NonRemovable           : False
Dependencies           : {Microsoft.VCLibs.140.00.Debug_14.0.27508.1_x64__8wekyb3d8bbwe,
                         Microsoft.NET.CoreRuntime.1.1_1.1.27004.0_x64__8wekyb3d8bbwe}
IsPartiallyStaged      : False
SignatureKind          : None
Status                 : Ok

XAML:
<Button cm:Message.Attach="GenerateRandomEquation" Margin="10,10,10,0" Content="Generate" HorizontalAlignment="Center" AutomationProperties.AutomationId="UIGenerateRandomEquationButton" />

Test class:
using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using OpenQA.Selenium.Appium.Windows;
using OpenQA.Selenium.Remote;

namespace XamlUwpApp.FunctionalTests
{
    [TestClass]
    public class UnitTest1
    {
        public TestContext TestContext { get; set; }

        [TestMethod]
        public void TestMethod1()
        {
            var capabilities = new DesiredCapabilities();

            capabilities.SetCapability("app", "52a95998-8cfa-43d7-96c9-06fc6d8ad1c5_<IP_ADDRESS>_x64__x38zh4nq6sjk4!app");

            var windowsDriver = new WindowsDriver<WindowsElement>(new Uri("http://<IP_ADDRESS>:4723/"), capabilities);

            windowsDriver.FindElementByAccessibilityId("UIGenerateRandomEquationButton").Click();
        }
    }
}

WinAppDriver response:
==========================================
POST /session HTTP/1.1
Accept: application/json, image/png
Connection: Keep-Alive
Content-Length: 126
Content-Type: application/json;charset=utf-8
Host: <IP_ADDRESS>:4723

{"desiredCapabilities":{"app":"52a95998-8cfa-43d7-96c9-06fc6d8ad1c5_<IP_ADDRESS>_x64__x38zh4nq6sjk4!app","platformName":"Windows"}}
HTTP/1.1 400 Bad Request
Content-Length: 110
Content-Type: application/json

{"status":100,"value":{"error":"invalid argument","message":"Value does not fall within the expected range."}}

Hi @nathan-alden-hp
Can you run get-StartApps via powershell and confirm the app name?

PS > Get-StartApps | ?{ $_.Name.Contains("XamlUwpApp") }

Name       AppID
----       -----
XamlUwpApp 52a95998-8cfa-43d7-96c9-06fc6d8ad1c5_x38zh4nq6sjk4!App

FWIW, I updated the Appium.WebDriver package to 4.0.0. I got some obsolete warnings, so I updated my test code to this:
var options = new AppiumOptions();

options.AddAdditionalCapability("app", "52a95998-8cfa-43d7-96c9-06fc6d8ad1c5_<IP_ADDRESS>_x64__x38zh4nq6sjk4!app");

var windowsDriver = new WindowsDriver<WindowsElement>(new Uri("http://<IP_ADDRESS>:4723/"), options);

windowsDriver.FindElementByAccessibilityId("UIGenerateRandomEquationButton").Click();

However, the driver still reports the same error.

Hi @nathan-alden-hp
Try passing in just " 52a95998-8cfa-43d7-96c9-06fc6d8ad1c5_x38zh4nq6sjk4!App" for the AppID - since it looks like there is a variation what is being reported.
If that doesn't work, then I'd recommend using a Desktop Session.

Hello @hassanuz,
I am facing the similar issue. Not able to launch my app through App ID.
Could you please suggest me the fix?
Thanks
Gaurav Agnihotri


UVI Request

--- UVI JSON ---
{
  "vendor_name": "Linux",
  "product_name": "Kernel",
  "product_version": "versions from v4.11 to before v5.12.13",
  "vulnerability_type": "unspecified",
  "affected_component": "unspecified",
  "attack_vector": "unspecified",
  "impact": "unspecified",
  "credit": "",
  "references": [
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=11538d039ac6efcf4f1a6c536e1b87cd3668a9fd",
    "https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/commit/?id=25053a8404ba17ca48f5553d487afc1882e9f56c"
  ],
  "extended_references": [
    {
      "type": "commit",
      "value": "11538d039ac6efcf4f1a6c536e1b87cd3668a9fd",
      "note": "introduced"
    },
    {
      "type": "commit",
      "value": "25053a8404ba17ca48f5553d487afc1882e9f56c",
      "note": "fixed"
    }
  ],
  "reporter": "joshbressers",
  "reporter_id": 1692786,
  "notes": "",
  "description": "net: bridge: fix vlan tunnel dst refcnt when egressing\n\nThis is an automated ID intended to aid in discovery of potential security vulnerabilities. The actual impact and attack plausibility have not yet been proven.\nThis ID is fixed in Linux Kernel version v5.12.13 by commit 25053a8404ba17ca48f5553d487afc1882e9f56c, it was introduced in version v4.11 by commit 11538d039ac6efcf4f1a6c536e1b87cd3668a9fd. For more details please see the references link."
}
--- UVI JSON ---

This issue has been assigned UVI-2021-1001090


MINOR: Bump trunk to 4.1.0-SNAPSHOT

Committer Checklist (excluded from commit message)

[ ] Verify design and implementation
[ ] Verify test coverage and CI build status
[ ] Verify documentation (including upgrade notes)

The Cut Branches step in the release process still mentions Jenkins. I'm not sure what changes are needed now with our new build pipeline.

@mimaison Opened https://github.com/apache/kafka/pull/18215 for the CI. It seems to me that we can use github actions now.


ValueError: too many values to unpack (expected 3)

~/aster.pytorch/lib/models/model_builder.py in forward(self, input_dict)
89       return_dict['losses']['loss_rec'] = loss_rec
90     else:
---> 91       rec_pred, rec_pred_scores = self.decoder.beam_search(encoder_feats, global_args.beam_width, self.eos)
92       rec_pred_ = self.decoder([encoder_feats, rec_targets, rec_lengths])
93       loss_rec = self.rec_crit(rec_pred_, rec_targets, rec_lengths)
~/aster.pytorch/lib/models/attention_recognition_head.py in beam_search(self, x, beam_width, eos)
74
75     # https://github.com/IBM/pytorch-seq2seq/blob/fede87655ddce6c94b38886089e05321dc9802af/seq2seq/models/TopKDecoder.py
---> 76     batch_size, l, d = x.size()
77     # inflated_encoder_feats = _inflate(encoder_feats, beam_width, 0) # ABC --> AABBCC -/-> ABCABC
78     inflated_encoder_feats = x.unsqueeze(1).permute((1,0,2,3)).repeat((beam_width,1,1,1)).permute((1,0,2,3)).contiguous().view(-1, l, d)
ValueError: too many values to unpack (expected 3)

I have not met such a problem but someone else has. I will reopen the related issue. Please refer to #6 to look for help.

Do you solve this problem?

I have find the reason， parameter  in config.py is different with scripts\stn_att_rec.sh.
Do not   run main.py  directly, run stn_att_rec.sh as readme show。


[Storage] Generate swagger for blobs (tag indexing for versions)

Tag indexing for versions feature work was completed previously but pulled from a previous release -- and consequently the generated files may be out of date. This PR aims to generate with the newest swagger to ensure the feature is ready for STG84 release.

API change check
APIView has identified API level changes in this PR and created following API reviews.
azure-storage-blob


else clause to send on error action

You can close this PR as I've added these changes to this PR (https://github.com/ConnectedHomes/ember-stickler/pull/3) which has been merged to master.


Error displaying when running the app

An error is displayed when I compile and run the web version or other platforms
"an unexpected error occurred"
what's wrong?

@abstruse98 Sorry you've been having trouble getting Outline to run. Can you copy and paste the commands you're running?

Closing due to inactivity. Please re-open if you would like to work through this.


Improve weekly spec drafts updating GitHub Actions workflow

Working with these is a bit annoying. I can never remember the steps needed to handle merging with main. I ended up making a few changes to the workflow file to try to make this more approachable. Changes include:

Running dist as part of npm run update-drafts. This is the single biggest headache I was having.
Explicitly deleting and adding files, so it's clearer what files this workflow is meant to update.
Using a label to identify generated PRs (this will make it easier for me to review these PRs alongside dependabot changes).
Using the peter-evans/create-pull-request to do the commit, instead of a custom step and committer config.
Linking to the workflow file from the pull request body.

OK, we'll find out how much I broke this on Monday. 😅

I found the first problem even sooner. https://github.com/web-platform-dx/web-features/pull/1735


Home page

Closes #5

Created new event card view for the home page carousel
Added a warning if the user has disabled JavaScript
Changed the strftime formatters to be platform agnostic
Added a template file for the keys json
Made the little info texts on the opportunities page smaller, so they don't break across lines on mobile
Styling tweaks and unification

I won't be able to fix it until later tonight, so we can go ahead and merge it, so Ryan can make sure it's all kosher with pythonanywhere. I'll make a new issue for the bug.


Add process name in the error message for easy debug

As different services log errors in the similar format [Error] it's hard to debug which service throws the error.
Add the service name as a prefix for the error message. Example [PSE][Error] This is an error

LGTM


Docker Compose Bug

How to reproduce the behaviour

docker-compose up

django_1    |     In file included from src/buffer.cpp:12: django_1    |     src/pyodbc.h:56:10: fatal error: sql.h: No such file or directory django_1    |      #include <sql.h>
Your Environment

Operating System: Mac Mojave 10.14
Python Version Used: 3.6
When you install doccano: 10/7/19
How did you install doccano (Heroku button etc): locally - clone the repo

After rebuilding the image docker built -t doccano . and adding the packages below into install-mysql.sh, and editing the docker-compose file to use image: doccano:latest, it worked.
apt-get update
apt-get install --no-install-recommends -y \
  curl=7.52.1-5+deb9u9 \
  build-essential \
  gnupg=2.1.18-8~deb9u4 \
  apt-transport-https=1.4.9 \
  unixodbc unixodbc-dev python-pyodbc libsqliteodbc```

related issue #374 . We have to confirm docker-compose.

Same issue.

OS: Manjaro.
Docker version 19.03.1-ce, build 74b1e89e8a
docker-compose version 1.24.1, build unknown
Cloned from git (commit 1066c3a)

Thank you for your contribution!!


Error: could not install smartsocket listener: Permission denied

Trying to run my Galaxy S9 on Ubuntu gives the following error:
* daemon not running; starting now at tcp:5037
ADB server didn't ACK
Full server startup log: /tmp/adb.1000.log
Server had pid: 20075
--- adb starting (pid 20075) ---
adb I 07-26 16:16:06 20075 20075 main.cpp:57] Android Debug Bridge version 1.0.39
adb I 07-26 16:16:06 20075 20075 main.cpp:57] Version 1:8.1.0+r23-5~18.04
adb I 07-26 16:16:06 20075 20075 main.cpp:57] Installed as /snap/scrcpy/136/usr/lib/android-sdk/platform-tools/adb
adb I 07-26 16:16:06 20075 20075 main.cpp:57]
error: could not install *smartsocket* listener: Permission denied

* failed to start daemon
adb: error: failed to get feature set: cannot connect to daemon
ERROR: "adb push" returned with value 1

I have enabled usb debugging on my device.

What is the result of
adb devices

?

List of devices attached
29e3b8dcca3f7ece	device

cc @sisco311

I followed the build the app manually and it worked! :smile:
I also had to install Android API 29 by Android Studio.
Thanks!

@rodrigocichetto I'm glad you managed to get it work. But it would be cool to find out why the snap did not work for you.
The error usually occurs when the snap is not connected to the network and/or network-bind slots of the core snap. This should happen automatically at installation time.
You can check the interface connections with:
snap connections scrcpy
If for some reason the snap is not connected to the above mentioned slots, you can connect it manually:
sudo snap connect scrcpy:network :network
sudo snap connect scrcpy:network-bind :network-bind
Of course the core snap must be installed on your system before you make the connections:
sudo snap install core


stopSubmit submit errors can get lost due to field resets

I have a login form with a saga based flow: on submit an action is dispatched which triggers an AJAX call to a backend. If the login failed an error action is dispatched, which should update the form to show the login failed. Since redux-form does not expose an action to set form errors directly I am doing this using stopSubmit. For some reason this works fine for the password field, but always fails for the email field.
Tracing the redux actions shows the following happens when I create an error for the password field:

STOP_SUBMIT is dispatched. form.login.submitErrors is updated correctly.
redux-form dispatches UNREGISTER_FIELD for the email field
redux-form dispatches REGISTER_FIELD for the email field

Interestingly when I do the exact same for the email field the flow is different:

STOP_SUBMIT is dispatched. form.login.submitErrors is updated correctly.
redux-form dispatches UNREGISTER_FIELD for the email field, which clears the submit error.
redux-form dispatches REGISTER_FIELD for the email field
redux-form dispatches UNREGISTER_FIELD for the password field
redux-form dispatches REGISTER_FIELD for the password field

I am not 100% sure if stopSubmit is the right action to use. updateSyncErrors feels like a better fit, but that is not exported.
Sandbox Link
https://codesandbox.io/s/jGxlWYL5

I thought #3211 was related, but I can still reproduce this with the 7.0.1 release.

This took me a moment... your sandbox is passing the email key for the username field 😆.

@danielrob My sandbox is trying to set both errors, but you only the password error is visible. Changing it to only set the email error will result in no error being shown at all.

I wasn't clear enough. Your sandbox is only setting the password error, and an error for the email field which is a non-existent field in your form. Your form contains username and password fields, not email and password fields.
The error is getting swallowed because of this key mismatch.

@danielrob You're right - that was an error in the sandbox. I do see that problem in my full codebase, and there is no mismatch there. I'll take another stab and reproducing that in the sandbox.

Because this issue saw almost no activity for a few months, I'm closing it.


Update Border router

Use Mbed OS coding style
Remove obsoleted configuration
Fix issue https://github.com/ARMmbed/nanostack-border-router/issues/140
Add build profile to Jenkinsfile
Free 32k RAM for 6LoWPAN BR by using malloc-for-heap flag

@mikter , @SeppoTakalo , @deepakvenugopal  would you please review?


Where is disableLoadMore?

where is disableLoadMore?
i want stop loadMore

The LoadMore method is disable in default. So if you don't want to use loading more you can simply remove enableLoadmore() method in demo.
You can remove these code in demo.
   ultimateRecyclerView.enableLoadmore();
   simpleRecyclerViewAdapter.setCustomLoadMoreView(LayoutInflater.from(this)
                .inflate(R.layout.custom_bottom_progressbar, null));

i know disable default....
my probleam mean...
i need enableLoadMore....
example 10 totalitem
call 1page 3item... loadmore 4 call after disableLoadMore need....
last page do not call loadmore

Thank you for giving this advice.
In fact ,the current released version do not have the method of disableLoadmore().But in the latest dev version it has been improved.
Please notice that the method has some side effects if you also use parallax head view in the same time.

Now you can use Ultimate Recyclerview version 0.3.1 to solve the problem.


Make k8s resource checks in http gateway optional

You only need to be paranoid about topic existence if you are running
in an actual cluster. Running locally (like it says you can do in the
docs) there is no need to check for topic existence in k8s because
there is no k8s.

I guess two comments after a brief skim.
First is we should be able to add a testcase that doesn't rely on providing a stub that fits the RiffTopicExistenceChecker interface to allow the tests to run outside k8s. Which is nice, it's a slightly more realistic test.
Second is that I'd prefer that we provide a second implementation of the interface that always gives "true" than to rely on nils to signal different policies. So instead of the alternatives being the current checker and nil, it'd be the current checker and AlwaysTrueExistenceChecker.

So it's less about detecting the k8s/non-k8s profile by reading the err of client.InClusterConfig(), which I think would remain as-is. It's that in the non-k8s case, setting the riffClient to nil, passing it forward and then nil-checking on the far side might be replaced by moving the creation of RiffTopicExistenceChecker into the k8s case and introducing a AlwaysTrueExistenceChecker for the non-k8s case.
Does that make more sense?

On second thoughts, the code looks like it will work and could be updated in a later PR, I'm going to pull it in.

Thanks. I'll refactor to AlwaysTrueExistenceChecker as an exercise.


PR: Fix all test files to be run independently

This also removes all code related to full path sorting, which was eliminated as part of PR #3946.

Hello @ccordoba12! Thanks for submitting the PR.

In the file conftest.py, following are the PEP8 issues :

Line 25:1: E402 module level import not at top of file

In the file spyder/app/start.py, following are the PEP8 issues :

Line 71:34: E261 at least two spaces before inline comment

In the file spyder/widgets/editortools.py, following are the PEP8 issues :

Line 321:9: E731 do not assign a lambda expression, use a def


Question: Kaggle kernels or Google Colab notebooks?

Great tool, look forward to using it and apply to my notebooks on https://github.com/neomatrix369/awesome-ai-ml-dl.
How would one run this on Kaggle Kernels or Google Colab notebooks? Can it be run from inside the notebook on the notebook? Or maybe we could use it via a Utility Script on Kaggle.
Would love to know? As this would open up a whole world of possibilities.

Hey, thanks for the question! I've been field-testing this with some Kaggle notebooks myself, will update with a guide in due course

I've added some notebooks to Kaggle, e.g. this one https://www.kaggle.com/marcogorelli/cln-titanic-data-science-solutions, which demos running some nbqa checks
Due to the nature of Kaggle notebooks, I don't think it's possible to do this on a running notebook, it looks like you need to download it, run nbqa locally, and then re-upload it.
Anyway, thanks for encouraging this idea, maybe with this being on Kaggle it'll spread more easily :smile:

Any update regarding this ? I have started to link my kaggle notebooks with github. I was wondering if there is a way to use this wihin kaggle or when submitting ? (I have opened a discussion regarding this on Kaggle)

hey - I'd suggest downloading the notebook, running whichever tool you want on it, and reuploading. with the kaggle cli, this is quite easy


Missing image in Docs

Description
Missing image in Changing Navbar's Logo, Title and Links at https://datahub.io/docs/Configuring+Nav+bar+and+SEO+fields

I thought I've fixed it already 😅


Added support for receiving inferred span children as OTLP span links.

Closes #175. Span-Links received via OTLP with the attribute elastic.is_child set to true will be stored as child.ids instead of links.
This is currently in draft until naming discussions about the attribute are settled.
I've manually tested this PR with our inferred-spans Java Otel extension:

We've discussed this in the APM agents weekly and decided to stay with the proposed elastic.is_child attribute name. I'll take care of also adding this to the agent specs.

@jackshirazi ran a program to test this and sent data to an APM Server BC5 in ESS; seeing the waterfall for the trace and the span:


Unable to create Euclid Flex Thin font

Unable to create Euclid Flex Thin font. I have used below code.
SKTypeface typeFace = SKTypeface.FromFamilyName(@"Euclid Flex", SKFontStyleWeight.Thin, SKFontStyleWidth.Normal, SKFontStyleSlant.Upright);
If we save the stream as a file, it shows the font name Euclid Flex Ultralight.
FileStream fontStream = new FileStream(@"D:\EuclidFont.otf", FileMode.Create, FileAccess.Write);
  SKStreamAsset stream = typeFace.OpenStream();
            if (stream != null && stream.Length > 0)
            {
                byte[] fontData = new byte[stream.Length - 1];
                stream.Read(fontData, stream.Length);
                stream.Dispose();
                MemoryStream m_stream = new MemoryStream(fontData);
                m_stream.CopyTo(fontStream);
                fontStream.Flush();
            }

SkiaSharp version - 1.59.3
NET Core 2.0

The SKTypeface.FromFamilyName() requires that the font be installed to the system (C:\Windows\Fonts)
If you don't have permissions or can't install, then reading from a stream is just fine. There are other options to load without copying:

SKTypeface.FromFile
SKTypeface.FromStream

Yes this font was already installed.

It might be because you are querying the font weight as SKFontStyleWeight.Thin or using SKFontStyleWidth.Normal.
I think this is a duplicate of #682

SKFontStyleWeight.Light is working for the same font. So, this issue occurs only for SKFontStyleWeight.Thin. Whether it is a known bug of SkiaSharp?

I'm closing this now because it is a duplicate of #682. Not a bug per se, but definitely a known limitation. This is something we can address in the upcoming v2 of the library.
I'll update the other issue with progress, so it might be good to subscribe to that one.


fix(navigation): Fixed links in Analyze-Overview

There were the following issues with your Pull Request

Commit: 7221b76694d7097447c2626b94122e2aa4f762f9
Your commit message does not contain a body

Guidelines are available at https://github.com/fabric8io/fabric8-ui

This message was auto-generated by https://gitcop.com

There were the following issues with your Pull Request

Commit: 8f2c3e6d1e0f6703916328289cd6e61ae45824bd
Your commit message does not contain a body

Guidelines are available at https://github.com/fabric8io/fabric8-ui

This message was auto-generated by https://gitcop.com


Remove Description of '+' Button from Plugin Homepage Prompt Text

The plugin homepage currently contains a prompt text that describes the '+' button located to the left of the input field. However, this button has been removed from the UI, and adding context can now only be done through the 'Add to DevChat' option available via right-click. This outdated description could confuse users about how to add context.
Suggested Change:

Please update the homepage prompt text to remove any mentions of the '+' button. It would help ensure the instructions align with the current UI and functionality.

Additional Context:

Adding context is an important feature for our users, and ensuring clear instructions on how to do so is crucial. The current primary method to add context is through the 'Add to DevChat' option by right-clicking, and this should be clearly communicated.

Thank you for addressing this to enhance user experience.

The feature has been merged into the GUI mainline, and updates can be seen upon dual-platform release.


Reduce maintenance difficulty

Hi there,
This project seems to be a little bit overwhelmed according to the recent popularity and the number of issue unanswered.
I would like to help you to drive contribution flow and ease project developement.
So I will create a bunch or PRs based on the following guide:
http://fitzgeraldnick.com/2018/01/30/foss-maintenance-stress.html
I really invite you to read this article because it can step up your project at another level.

Hi! Your help and advices are very valuable. Thank you!
Really, I want to keep this project maintained. And I haven't enough time to respond to everyone, it's true. But I still add important changes to the code and this project still maintained.
I want to create a group of contributors who could help me to make this project more welcoming and help me to solve gathered issues.
I think I should create a group and send invites to all potential maintainers soon. 👍

I post some documentation about github labels.
I think labels will help you organize issues.
https://github.com/Relequestual/sensible-github-labels
https://github.com/FakeItEasy/FakeItEasy/wiki/Issue-labels
https://robinpowered.com/blog/best-practice-system-for-organizing-and-tagging-github-issues/

I'd be happy to include this crate in https://github.com/rust-crates if you would like.
A "rust-web" group might be good as well.

Has this been resolved now?

@teymour-aldridge Yes it is! Thanks for the reminder.


Investigate receiving input without using window event flow

Using window events can really suck if something on the OS decides to block the macOS main thread. Any app can do this and it will cause a delay in receiving events across all apps (example https://github.com/ppy/osu/issues/26604). I've hit this many times myself.
It may also be relevant on windows for key input, as we are not using raw input there (I've seen reports of some users hitting input delays on windows with similar blocking).
cc @Susko3 since you might have something to add here.

If I understand correctly, blocking the main thread only happens because we (or SDL) asks the OS for events (on Windows that would be via PeekMessage()) and that call blocks / takes a while for whatever reason. So the idea is to get (specifically) input events in another thread?
Would be useful to have a program that can generate this kind of lag. I played around in AutoHotkey and the best lag generator I could come up with is just sending a lot of mouse clicks. I'm unsure how representative this is of real-world lag, but it does spike the InputThread WndProc monitor and makes keyboard and mouse input lag.
Anyways, SDL3 recently added support for pooling Windows raw mouse input in a separate thread (https://github.com/libsdl-org/SDL/issues/8756).  I believe it should be possible to hook directly into these events by installing an SDL event filter/watch, like we currently do for other specific events.
It probably isn't too hard to add RAWKEYBOARD handling to SDL.

It may also be relevant on windows for key input, as we are not using raw input there (I've seen reports of some users hitting input delays on windows with similar blocking).

onWndProc in WindowsMouseHandler is only called by SDL_PumpEvents(). So it is still very much tied to the event loop.

SDL3 has added raw keyboard on windows: https://github.com/libsdl-org/SDL/pull/9363. This is starting to look like a really interesting update.


Added devcontainer and initial Dockerfile

This way, developers can simply clone this repo in VS Code or open a GitHub Codespace, and the development environment will be set up automatically

Woops, I didn't know GitHub would automatically close my PR if I deleted my fork. I'll make a new PR


Delete account via URL

Google Play requires a way to delete your account.  Has to be on web.  So, I'm thinking a delete page, user & password, a I'm sure I want to delete my account button, and a "THIS WILL DELETE YOUR ACCOUNT, ALL ITEMS, BADGES, XP, and CANNOT BE UNDONE" button after that.  Server side will need to understand how to wipe an account.  I should be able to do the web portion once there's some hooks.
  Not sure how you do this but those will have to be wiped too.

Instead of deleting chips, we'll dump them all instead.  Way cooler.

Updated issue.  Anonymized data should be kept as not to break things, as well as for analytics.

Moving to https://github.com/SitekickRemastered/Web/issues/15


HTML attributes being eliminated - Customizing Colors in Markdown Widgets not working

Screenshot
[drag & drop image(s) here!]

Description
[describe the issue here!]
Following a basic markdown styling as bellow:
https://www.restack.io/docs/superset-knowledge-superset-markdown-color-guide
The only attribute being accepted is ID and it is changed when rendered. For instance,

Firefox

the only way for that to work is by live css and use the id "user-content-XXX" that is auto generated based on my id
#user-content-teste {
  color: red;
}

All other atributes are eliminated.
Design input
[describe any input/collaboration you'd like from designers, and
tag accordingly. For design review, add the
label design:review. If this includes a design proposal,
include the label design:suggest]

In your config.py file, you can configure which attributes you want to allow on which tags. It's locked down by default for security purposes, but if you want to punch holes, that's up to you :)
It's right around these comments, where you can see some example config:
# HTML_SANITIZATION_SCHEMA_EXTENSIONS = {
#   "attributes": {
#     "div": ["style"],
#   },

I'll close this out in hopes it answers your question, but we can reopen if there's more to deal with :D


double or more thai vowel are not show in VSCode editor

Issue Type: Bug

Open VSCode
new File , then type any thai double thai vowel e.g. { ่, ้, ๊, ๋ }
In editor are combine in one vowel but file are save correctly.

VS Code version: Code 1.66.2 (Universal) (dfd34e8260c270da74b5c2d86d61aee4b6d56977, 2022-04-11T07:49:20.994Z)
OS version: Darwin x64 21.3.0
Restricted Mode: No
<!-- generated by issue reporter --
<img width="836" alt="Screen Shot 2565-04-25 at 10 16 32" src="https://user-images.githubusercontent.com/104336867/165015010-b9def4fc-1fbd-447f-97d6-0a0eac5cb49a.png">
>

@SajjaAtikij Could you please provide an example word which uses these characters that I can copy-paste? How does this word render in TextEdit? How does it render in vscode?

example word are below:-
ดีีีีีี
ปีีีีีี
ที่่่่่่

In Text editor and search bar in VSCode: multiple thai vowel (upper character) see picture
https://user-images.githubusercontent.com/104336867/165156955-7cee7f81-eada-4590-92e5-b0c7ce07d3d1.png

In VScode editor  (macOS): combine in one vowel see picture
https://user-images.githubusercontent.com/104336867/165157334-f21b8e7b-d510-49b4-86b5-3d22f09959fa.png

This is controlled by the font that is used. e.g. configuring "editor.fontFamily": "Arial":

"editor.fontFamily": "Fira Code":

I can recommend to switch to a font that better renders these characters if monospace fonts are not good enough.


Unknown at rule causes next at rule to highlight entire document

Steps to Reproduce:

create two identical at rule rules. Placing your cursor on the "@___" part of one rule normally highlights all identical rules in the document,

Change one of the at rules to an unknown value. Now the entire document gets highlighted when you place your cursor on the "@___" part of any rule below the modified one.

Actually looks like placing your cursor on CSS tokens anywhere inside or beneath the unknown rule causes the strange highlighting.

VS Code version: Code - Insiders 1.25.0-insider (f2fa6bea46bde9856e6a593e4d4c5af2b303ac66, 2018-06-27T05:18:12.763Z)
OS version: Windows_NT x64 10.0.16299

#52864

Closing this in favor of a more general issue.


Support streaming with NO_COMPRESSION_64 and NO_COMPRESSION_32

Great library and easy to use API. However, I was a bit disappointed to discover that NO_COMPRESSION_64 and NO_COMPRESSION_32 options don't produce a streaming zip. For a library called stream-zip, having an option that, in fact, does not really produce a streaming zip seems a bit unexpected ;) (I must admit I should have read the documentation more clearly, but this is still very surprising, given the purpose of this library)
I am looking for a library to produce large streaming zip which packages other files with STORE instead of DEFLATE, and assumed the NO_COMPRESSION_* options would do so.
It's a bit of an usual chance, as uncompressed streaming seems easier than compressed streaming.
Would you be willing to add support for uncompressed STORE-mode streaming?
I assume it's not supported because the crc32 and size are stored in the local header before the data.. (edit: I see the data descriptor option is used for compressed data)
The way other tools handle this, for example client-zip, is to simply store 0 for the size and crc32 in the local record: https://github.com/Touffy/client-zip/blob/master/src/zip.ts#L124
and store the real values in central directory record which is computed after the stream is read.
I think this seems to work fine most tools, since the central directory is the main data used, and I think would be preferable to buffering the entire file - especially for zip64 since the use most likely is dealing with a very large file if requesting zip64 format.
Thanks again and happy to help test this!

Create a quick patch that does uncompressed streaming, this works well for our case:
https://github.com/uktrade/stream-zip/compare/main...ikreymer:stream-zip:stream-uncompress?expand=1
happy to also do same for zip32 and open a PR if you'd like!
The resulting zip is readable and considered valid by unzip -t

Would you be willing to add support for uncompressed STORE-mode streaming?

Maybe... but...

store 0 for the size and crc32 in the local record:

The main problem is that I think I think it makes the ZIP not stream unzip-able, which is a bit of a shame. Streaming unzippers I think need some way of knowing when they get to the end of the data. Without the size in local header, they can't know this for sure(?)
(Compressed data with DEFLATE doesn't need this since it indicates within the data stream when it reaches the end)
But... there might be a way of doing this. As of quite recently, there is the concept of "dynamic" modes, and specifically ZIP_AUTO. It accepts parameters. We could have something similar for NO_COMPRESSION_* that accepts the size and crc32? It might be a bit of a faff in situations to find these ahead of time, especially the crc32, but at least it makes it possible.

The main problem is that I think I think it makes the ZIP not stream unzip-able, which is a bit of a shame. Streaming unzippers I think need some way of knowing when they get to the end of the data. Without the size in local header, they can't know this for sure(?)

That's a fair point, if the goal is to be able to always stream unzip with stream-unzip. In general, I think ZIP files can be stream zipped and random-access unzipped (this is all our use case), but stream unzipping is more problematic (eg: https://github.com/thejoshwolfe/yauzl#no-streaming-unzip-api)

(Compressed data with DEFLATE doesn't need this since it indicates within the data stream when it reaches the end)
But... there might be a way of doing this. As of quite recently, there is the concept of "dynamic" modes, and specifically ZIP_AUTO. It accepts parameters. We could have something similar for NO_COMPRESSION_* that accepts the size and crc32? It might be a bit of a faff in situations to find these ahead of time, especially the crc32, but at least it makes it possible.

What about adding a NO_COMPRESSION_STREAM_64 and NO_COMPRESSION_STREAM_32 that would stream zip but not be stream unzippable? That way, wouldn't break backwards compatibility.
Another option is to add 2 optional properties, so could include size and crc32 in addition to the 5 require properties? If at least the size is provided, then could still do the stream unzipping (though, I guess that wouldn't be spec conformant still..). Perhaps ZIP_AUTO could then choose NO_COMPRESSION_STREAM_* or NO_COMPRESSION_* based on whether the 6th size property is provided..

That's a fair point, if the goal is to be able to always stream unzip with stream-unzip.

Yes, but there are other stream unzippers as well. e.g. https://github.com/madler/sunzip and libarchive that you can call from Python using, for example, ctypes as in https://stackoverflow.com/a/74986842/1319998

Another option is to add 2 optional properties, so could include size and crc32 in addition to the 5 require properties?

I think fairly not keen on this approach - currently the 5 properties are always required and used, so it feels odd a bit.

What about adding a NO_COMPRESSION_STREAM_64 and NO_COMPRESSION_STREAM_32 that would stream zip but not be stream unzippable?

I'm torn... it adds another layer to the whole "what zip clients can open the zip" thing. The zip32/zip64 situation is complex enough...
Just to check though, are you especially not pro using ZIP_AUTO with level=0? That's very close to stored, but yes, a bit of overhead. And is it definitely not possible for you to determine size and crc32 ahead of time?

Also on:

but stream unzipping is more problematic (eg: https://github.com/thejoshwolfe/yauzl#no-streaming-unzip-api)

From the link I see this:

However, the Local File Headers are explicitly denounced in the spec as being unreliable copies of the Central Directory

I didn't realise this, although quickly looking through https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT now, it doesn't seem to really denounce it from what I can tell. And the ISO version https://nfoworks.org/notes/2013/04/n130401-13R5EoW9HY4ZvQH/n130401c11-ISO_IEC_21320-1_2015.pdf doesn't really seem to do that either from what I can tell.

Just to check though, are you especially not pro using ZIP_AUTO with level=0?

Or... one of the other ZIP_ modes passing in a custom zlib object with level=0

What about adding a NO_COMPRESSION_STREAM_64 and NO_COMPRESSION_STREAM_32 that would stream zip but not be stream unzippable?

Just to communicate, I am more and more leaning to not having a method where the resulting zip cannot be stream unzipped. The downsides of adding such a method I think are:

I fear it's too much of a gotcha. Just like in this issue actually, I fear people not reading the docs, and getting bitten by it after ZIP files are generated, which is a bit late.
It then adds to the list of trade offs you have to make when using the library if you do read the docs.
And it's not one of those things where you can tell the client "oh just upgrade to a more recent version" - using a stream unzipper is a sort of core architectural choice.

While there are alternatives that I think avoid these:

Passing "level=0" is an option right now which will disable compression in a stream unzip friendly way, at the cost of a few extra bytes (I think something like 1 extra bytes every 10k? It's quite low...). So I think a reasonable work around exists now for most cases. If there was some client that didn't support DEFLATE, or if the overhead was bigger, I might have a different opinion, but I think it's a very small overhead and I know of no such client.
Adding a dynamic method of: NO_COMPRESSION_32(uncompressed_size, crc_32) say would cover a few more cases. Although I suspect crc_32 might be a bit annoying to calculate sometimes. But if files are stored on S3 say, then it looks like CRC32 can be added as a trailer https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutObject.html#API_PutObject_RequestSyntax on upload to S3, and so still calculated on the fly. I don't think I've seen an HTTP trailer used in the wild though...so don't know if there are some issues here. And of course, not everyone uses S3!

But yes... the case of wanting to stream write a zip file with STORED members without knowing both the uncompressed size and crc32 ahead of time, that would be left essentially as a wontfix. But: am curious as to what the specifics of the consequences of this would be though - what pain does it cause, what does it make difficult or impossible etc.

But yes... the case of wanting to stream write a zip file with STORED members without knowing both the uncompressed size and crc32 ahead of time, that would be left essentially as a wontfix. But: am curious as to what the specifics of the consequences of this would be though - what pain does it cause, what does it make difficult or impossible etc.

We have a packaging format that uses ZIP to package smaller files into a distributable package. The format is accessed via random access: seek to read the central directory, then get file offsets in the zip, then offsets into each file from the beginning. Using STORE, we can seek from beginning of the file offset to get the data. Using DEFLATE with compression 0 could in theory work, but would have to change the format to also support this, and readers of the format would need to support both STORE and DEFLATE level 0, which just complicates the format spec and provides no benefit since the data must be uncompressed for it to work at all. I don't think its an option to use DEFLATE level 0 for this reason.

I fear it's too much of a gotcha. Just like in this issue actually, I fear people not reading the docs, and getting bitten by it after ZIP files are generated, which is a bit late.

I very much disagree, as I was bitten by the current behavior! The library says 'Python function to construct a ZIP archive on the fly - without having to store the entire ZIP in memory or disk.' and it has an option called NO_COMPRESSION_64, so I thought it would do exactly as advertised. On the contrary, the library does the exact opposite -- it in fact stores the entire file in memory. Worse, if you're asking for zip64, chances are, you know you're like dealing with very large files (in our case, likely tens or hundreds of GB), which the library is more than happy to buffer entirely into memory!
Not everyone who finds this library needs streaming unzip, but probably they are looking for streaming zip functionality w/o unbounded buffering of large files.
I think having options to do both would perhaps most helpful for different use cases.

The format is accessed via random access: seek to read the central directory, then get file offsets in the zip, then offsets into each file from the beginning. Using STORE, we can seek from beginning of the file offset to get the data.

Oooh... I have to admit, I never considered random access within a member file. Tricky to do this with DEFLATE.

I very much disagree, as I was bitten by the current behavior!

Ah I more mean bitten by making ZIPs that can't be stream unzipped. It sounds like this isn't important for you, but I don't want to go and introduce gotchas for others.
Will continue to ponder...

Adding a dynamic method of: NO_COMPRESSION_32(uncompressed_size, crc_32)

Would you be able to use such a dynamic method so we can have our cake and eat it too? It would mean the file can be both stream written and stream read...

One thing I'm curious about... On the wacz format, it looks like from https://specs.webrecorder.net/wacz/1.1.1/#zip-compression

All archive/ files should be stored in ZIP with 'STORE' mode.

But from https://specs.webrecorder.net/wacz/1.1.1/#archive

The archive directory MUST contain one or more files in the [WARC] format. The files SHOULD use the .warc file extension unless they are GZIP encoded in which case they MUST use the .warc.gz file extension.

So it looks like readers have to support not being able to seek into the STORED file anyway, since they have to support the STORED file being gz-encoded?

Actually more curiosity - can the warc files inside the zip themselves made in a streaming way?

Adding a dynamic method of: NO_COMPRESSION_32(uncompressed_size, crc_32)

Have done this in https://github.com/uktrade/stream-zip/pull/63

I know I'm not waiting for your answers, but I've pondered a bit more now...
... And although I see that it is useful in some circumstances to make STORED members in a streaming way with no up-front knowledge of the size and CRC 32, I'm going to treat that as beyond the scope of this project, because doing so makes ZIP files that can't be stream unzipped. The scope has to be limited somewhere, and I think I'm pro the line being drawn at adding behaviour that makes the files generated not stream unzippable.
I understand that might not be great news for your use case, but I'm happy to advise on what there is in stream-unzip, how to find the size and/or CRC 32 say to use NO_COMPRESSION_32(uncompressed_size, crc_32) (which might be trivial, but you never know...), or to give some thoughts on some other workaround.

That is a bit disappointing, as we'll have to use a fork, which works of course or find a different library to do this...
I would then suggest removing 'NO_COMPRESSION_64' and only having 'NO_COMPRESSION_64_BUFFERED' and 'NO_COMPRESSION_64_STREAMING' (and probably same for 32) so user does get bitten by unexpected behavior. I can't imagine a situation where a user would want to buffer a 4GB+ file so that should definitely not be a default behavior for NO_COMPRESSION_64 imo.
The WARC files are themselves gzip compressed, but as far as ZIP is concerned, they're added as STORE, because they're just files packaged into the ZIP. We're combining two formats: WARC and ZIP to create a package (WACZ) that is designed to be random-access readable. WARCs themselves are not random-access readable but can streamed (they're just concatenated gzip members). The point of this format is to provide a random-access readable format using ZIPs + their contents. I see you have a sqlite-s3-query implementation - the idea is very similar, we only use HTTP range requests to read the data from a potentially very large zip! That's why the streaming trade-off that you've proposed doesn't really make sense for us - the data is only ever read via random access and never stream-unzipped.

we'll have to use a fork, which works of course or find a different library to do this...

Ah understood

WARCs themselves are not random-access readable but can streamed (they're just concatenated gzip members). The point of this format is to provide a random-access readable format using ZIPs + their contents

I can't help but wonder now... since it seems that WACZ allows other files in the ZIP as DEFLATE, then you need some sort of HTTP-range friendly ZIP reader that supports DEFLATE members. So it's not a meaningful simplification to forbid DEFLATE on the things that aren't random access readable anyway?

I can't imagine a situation where a user would want to buffer a 4GB+ file so that should definitely not be a default behavior for NO_COMPRESSION_64 imo.

Very true

I would then suggest removing 'NO_COMPRESSION_64' and only having 'NO_COMPRESSION_64_BUFFERED' and 'NO_COMPRESSION_64_STREAMING' (and probably same for 32) so user does get bitten by unexpected behavior.

Actually I'm now wondering whether to completely remove NO_COMPRESSION_64 and NO_COMPRESSION_32

Users can roll their own versions using NO_COMPRESSION_64(uncompressed_size, crc_32) and NO_COMPRESSION_32(uncompressed_size, crc_32) respectively
They introduce gotchas
It'll keep this library more minimal
And specifically, all supplied modes would be streaming. Any non-streaming-behaviour in order to find the size or crc32 would be out of this library.

Ah but if the user already has the binary contents in a single bytes instance, contents say, then much simpler:
chunks = (contents,)
uncompressed_size = len(contents)
crc_32 = zlib.crc32(contents)

The pain it causes: my users often download archives comprised of files > 50GB with a total size > 1TB. The zip file keeps them in a single file, organized by internal folders.
We need to be able to handle files up to the limit of S3 keys and zip files up to their 5TB/ 16EB limits respectively, which requires that we buffer nothing more than is necessary on the server, and produce a valid archive. I don't require that it be stream-unzippable and I can't handle the overhead that making it so would require. I need to stream in keys and stream out a zip file.

@mdineen

I need to stream in keys and stream out a zip file.

So you can do this right now with stream-zip:

With the ZIP_* methods. If you don't want to compress, you can still avoid it using https://stream-zip.docs.trade.gov.uk/advanced-usage/#custom-zlib-options. But yes: there is a small amount of overhead in the final output. I can't remember exactly, something like 3 bytes of overhead every 64k bytes, + 1 byte? Something like that...

With the NO_COMPRESSION_* methods that accept a length and CRC32. The length in your case you can get from S3... but yes, the CRC32 you would have to have pre-compute.

Are either of these feasible in your case?

I will try this
Not preferable to calculation on demand

we want to avoid unnecessary compute for files that won't be downloaded in a zip
io-bound CRC32 task process will delay archive availability until all checksums are available

@mdineen

Not preferable to calculation on demand

Yeah it's awkward...

my users often download archives comprised of files > 50GB with a total size > 1TB
[...] We need to be able to handle S3 keys and zip files up to their 5TB/ 16EB limits respectively

I am curious - are you able to share some details on your use case/situation?

We're serving the legal sector with a filesharing platform with no size limits. User A will upload thousands of files ranging from a few KB to 100s of GB in an app-defined folder structure. User B will then download the entire folder tree as a zip file. We use the zip format to maintain the folder structure and to allow user B to keep the entire archive as a single file.
A single matter could span years and still require that an uploaded folder tree be downloadable as an archive without having to compute checksums in advance. Using EC2 to pre-calculate a checksum on an S3 keys takes ~30s/GB. All that IO is unnecessary when the streaming code is already reading all the bytes and the CRC32 is stored after. My test archive is 250 GB with 5 files @ 50 GB each, so I need over 2 hours of precomputation time to cache the crc32s. That doesn't scale.

Ah thanks for the details. All good to know.
So I would think the ZIP_64 or ZIP_AUTO methods should be able to handle those cases. Again just for curiosity/for the future really - are you deliberately wanting to avoid compressing the files in this case, or does it not matter?

zip files up to their 5TB/ 16EB limits respectively
User B will then download the entire folder tree as a zip file.

One thing I'm again curious about here: the users have  ~16EB of space locally to store a single file, and are able to use it? My world is... considerably smaller...

so I need over 2 hours of precomputation time to cache the crc32s. That doesn't scale.

The one thing that crosses my mind here is the files must get to S3 somehow? Can the crc32s be computed on the way in? (With a one time job for anything that's already there). It's not obvious to me that this can't be done in a way that would be as scalable as getting the files into S3 in the first place.

All that IO is unnecessary when the streaming code is already reading all the bytes and the CRC32 is stored after

Just for completeness really so everyone, including people looking at this later, are on the same page: the crc32 is stored after the data for the ZIP_* methods, but before the data for the NO_COMPRESSION_* methods.

We have avoided compression to ensure that there is no scalability limit based on CPU. Clients' disk space is at less of a premium than AWS EC2 time.
In practice the maximum archive size has been ~1TB. Mean size is closer to 180 GB. The solution we're looking for should ideally be scalable to the limit of the zip file specification. We're trying to replace nginx mod_zip, which scales up to these limits without appreciable memory or CPU overhead.
We've used cryptjs in our web app to capture the crc32 of the file at upload time, which could be an effective method going forward but makes legacy file sources and the decade of client data we have stored so far unusable.
Thanks for that clarification. Going to implement a test with ZIP_AUTO and post the comparison to my test with a node/express candidate written for this project as well.

As promised, here is my app.py for a uWSGI runtime:
from datetime import datetime
from stream_zip import stream_zip, ZIP_64
import boto3
import zlib

def application(env, start_response):
    now = datetime.now()

    # AWS S3 Configuration
    s3_client = boto3.client('s3', aws_access_key_id=KEY_ID, aws_secret_access_key=SECRET_KEY)
    bucket_name = BUCKET

    # List of S3 keys to be zipped
    s3_keys = [
        'dd/c/ed3e6aed-e63e-438e-997e-503b599e0ddc',
        '61/5/59f9e614-d7dd-456a-adcf-48ed97ee2615',
        '71/9/b9fae65c-8a97-4841-8158-e10447f71719',
        'f4/a/bbf8a239-9102-4093-a3f6-7c1b110cdf4a',
        'ec/9/5f4d9b86-94f0-43fb-a8b7-4e21991daec9',
    ]

    # Retrieve the streams from S3
    def member_files():
        for key in s3_keys:
            yield (key, now, 0o600, ZIP_64, s3_client.get_object(Bucket=bucket_name, Key=key)['Body'])

    # Set the headers
    headers = [('Content-Type', 'application/zip'),
               ('Content-Disposition', 'attachment; filename="large_file.zip"')]
    start_response('200 OK', headers)

    # Stream the zipped content
    get_compressobj = lambda: zlib.compressobj(level=0)
    return stream_zip(member_files(), get_compressobj=get_compressobj, chunk_size=512*1024*1024)

Using the 512MB buffer demonstrates tha this code is buffering the chunks, and the pattern of changes during the download says the same. The default chunk size of 64k is much too small for S3.

Am I missing anything?

The only thing that leaps out at me is this:

The default chunk size of 64k is much too small for S3.

I'm surprised this affects it - I think I have fetched from S3 and zipped the results with the default chunk size fine. By my understanding, S3 shouldn't even be aware of it... at least directly.
Potentially it would be aware indirectly because it could affect performance of the Python code, so affects how fast it pulls bytes from the socket, which would then affect the speed that S3 sends its bytes/packets on a TCP level. (I have to admit I don't fully know what I'm talking about here...)
What happens when you leave it as the default?

Ah one other thing... I have experienced some odd issues with ZIP_64 and Safari...

Safari unzips ZIP files by default (which is annoying at the best of times...)
But when the first file in a ZIP is ZIP_64, it seems to just unzip one file and that's it. The others are lost

Maybe I'm out of date admittedly

What happens when you leave it as the default?

The download speed reduces to < 100 KB/s. It seems to make an API request for every chunk, each of which has headers and gets TLS'd and authenticated by S3. 250GB / 64KB = 4.2 M API calls.

Safari unzips ZIP files by default (which is annoying at the best of times...)

https://discussions.apple.com/thread/253077187?sortBy=best

But when the first file in a ZIP is ZIP_64, it seems to just unzip one file and that's it. The others are lost

If you're certain those zips aren't corrupt, I recommend a different tool. The OS file browsers support ZIP, but in the size ranges we're creating they're flaky. We use 7-zip on Windows 11 to test the large zips, which is very effective.
The way this works with the 512 MB chunk size is quite choppy:
https://github.com/uktrade/stream-zip/assets/1494976/67212250-2e5f-409b-9728-d258f992bfa0

What happens when you leave it as the default?

It seems to make an API request for every chunk

Woah... nothing in my mental model suggest that this is possible. All stream-zip is doing it iterating over the iterable of bytes instances that are passed to it, which in this case is s3_client.get_object(Bucket=bucket_name, Key=key)['Body']. So a slightly more manual version of this:
for chunk in s3_client.get_object(Bucket=bucket_name, Key=key)['Body']:
    # do stuff

And then chunk_size controls subsequent processing of each chunk - how they are split or aggregated. So for that processing to then result in a new HTTP request to S3... there would have to be something very magical going on.

https://discussions.apple.com/thread/253077187?sortBy=best
If you're certain those zips aren't corrupt, I recommend a different tool. The OS file browsers support ZIP, but in the size ranges we're creating they're flaky. We use 7-zip on Windows 11 to test the large zips, which is very effective.

Oh yeah - just wanted to warn in case your users used Safari to download the files you're making - I guess I think in a lot of cases telling users to change their browser settings is not a good option.

The way this works with the 512 MB chunk size is quite choppy:

I would expect quite a lot of choppyness with such a chunk size because it would gather 512MB, output it to the user, and then only gather 512MB. There would be very little effective parallelism between getting stuff from S3, and sending it to the user as well because I think the various buffers involved would be way way smaller than 512MB

What happens when you leave it as the default?

The download speed reduces to < 100 KB/s

Interesting/surprising... I guess I would be curious as to what happens if you reduce it further, even to like 8k

The download speed reduces to < 100 KB/s

Or... what happens if you remove the web app part of it, and just run it from a Python script?

The issue appears to be with how boto fulfils stream requests. Rather than returning the streaming http response for the whole key, it create a request for each range (chunk) requested.
Trying a somewhat different approach using boto to sign, and requests to download. It looks to provide the exact behaviour we want and makes only 1 API request per file.
from datetime import datetime
from stream_zip import stream_zip, ZIP_64
import io
import boto3
import requests
import zlib

def application(env, start_response):
    now = datetime.now()

    # AWS S3 Configuration
    s3_client = boto3.client('s3', aws_access_key_id=KEY_ID, aws_secret_access_key=SECRET_KEY)
    bucket_name = BUCKET

    sign_url = lambda key: s3_client.generate_presigned_url('get_object', Params={'Bucket': bucket_name, 'Key': key}, ExpiresIn=6000)

    # List of S3 keys to be zipped
    s3_keys = [
        'dd/c/ed3e6aed-e63e-438e-997e-503b599e0ddc',
        '61/5/59f9e614-d7dd-456a-adcf-48ed97ee2615',
        '71/9/b9fae65c-8a97-4841-8158-e10447f71719',
        'f4/a/bbf8a239-9102-4093-a3f6-7c1b110cdf4a',
        'ec/9/5f4d9b86-94f0-43fb-a8b7-4e21991daec9',
    ]

    # Retrieve the streams from S3
    def member_files():
        for key in s3_keys:
            url = sign_url(key)
            yield (key, now, 0o600, ZIP_64, get_s3_stream(url))

    # Set the headers
    headers = [('Content-Type', 'application/zip'),
               ('Content-Disposition', 'attachment; filename="large_file.zip"')]
    start_response('200 OK', headers)

    # Stream the zipped content
    get_compressobj = lambda: zlib.compressobj(level=0)
    return stream_zip(member_files(), get_compressobj=get_compressobj)

def get_s3_stream(presigned_url):
    with requests.get(presigned_url, stream=True) as response:
        response.raise_for_status()
        for chunk in response.iter_content(chunk_size=8192):
            yield chunk

Please note I'm yielding 8k chunks here, and it's working very well so far.

Ah good!

The issue appears to be with how boto fulfils stream requests. Rather than returning the streaming http response for the whole key, it create a request for each range (chunk) requested.

Can I ask how you determined this? Is this documented anywhere? I know S3 and boto supports range requests, but I thought you had to be explicit.

Can I ask how you determined this? Is this documented anywhere? I know S3 and boto supports range requests, but I thought you had to be explicit.

If you turn on debug logging for boto you can see the requests in the logging. We first discovered this working with pypdf using automatic page detection, which reads the stream one byte at a time. Every read(1) to the body creates a few kb of api traffic.
logging.basicConfig(level=logging.DEBUG)
boto3.set_stream_logger('boto3.resources', logging.DEBUG)
boto3.set_stream_logger('botocore', logging.DEBUG)

s3 = boto3.client('s3')
response = s3.get_object(...)

Initial tests look very good! Once the implementation is cleaned up I'll submit it for your examples if it would be helpful.

Can I ask how you determined this? Is this documented anywhere? I know S3 and boto supports range requests, but I thought you had to be explicit.

If you turn on debug logging for boto you can see the requests in the logging.

So just done this...
import logging
import boto3

logging.basicConfig(level=logging.DEBUG)
boto3.set_stream_logger('boto3.resources', logging.DEBUG)
boto3.set_stream_logger('botocore', logging.DEBUG)

s3 = boto3.client('s3')
response = s3.get_object(
    Bucket='...'
    Key='...'
)
for chunk in response['Body']:
    print(len(chunk))

and for good measure one with manual read calls
import os
import logging
import boto3

logging.basicConfig(level=logging.DEBUG)
boto3.set_stream_logger('boto3.resources', logging.DEBUG)
boto3.set_stream_logger('botocore', logging.DEBUG)

s3 = boto3.client('s3')
response = s3.get_object(
    Bucket='...',
    Key='...',
)
for chunk in iter(lambda: response['Body'].read(1024), b''):
    print(len(chunk))

and in both I see loads of initial botocore and urllib3 logs, showing intial HTTP requests. But then just a load of:
1024

until it ends. Which is what I expected really - the response of a single HTTP request is streamed out, and no evidence of a API request per chunk. Also did it with read(1) and that's similar - no API calls intermingled.

Once the implementation is cleaned up I'll submit it for your examples if it would be helpful.

I think I would need to really understand why it's needed over just using boto3 more directly, e.g. see boto3 not streaming out a request/response

What happens when you leave it as the default?

The download speed reduces to < 100 KB/s

I can't reproduce this anymore. After removing the size arg the download is working smoothly and very fast.
Sorry for misreporting this. While creating the test code I iterated on the member_files generator a few times and could have been closing the stream on completion. On the bright side, I have a great reference implementation that supports huge files, plus I can use S3 keys or http endpoints as file sources.
-     return stream_zip(member_files(), get_compressobj=get_compressobj, chunk_size=512*1024*1024)
+     return stream_zip(member_files(), get_compressobj=get_compressobj)

Sorry for misreporting this

All good! Please keep the reports coming

On the bright side, I have a great reference implementation that supports huge files

It's good to know for me too - I know stream-zip works enough for our use cases/sizes, but it's hard to get an idea of what other people are using this for, what sort of sizes/constraints they're working with, that sort of thing.

The issue appears to be with how boto fulfils stream requests. Rather than returning the streaming http response for the whole key, it create a request for each range (chunk) requested.
Trying a somewhat different approach using boto to sign, and requests to download. It looks to provide the exact behaviour we want and makes only 1 API request per file.
from datetime import datetime
from stream_zip import stream_zip, ZIP_64
import io
import boto3
import requests
import zlib

def application(env, start_response):
    now = datetime.now()

    # AWS S3 Configuration
    s3_client = boto3.client('s3', aws_access_key_id=KEY_ID, aws_secret_access_key=SECRET_KEY)
    bucket_name = BUCKET

    sign_url = lambda key: s3_client.generate_presigned_url('get_object', Params={'Bucket': bucket_name, 'Key': key}, ExpiresIn=6000)

    # List of S3 keys to be zipped
    s3_keys = [
        'dd/c/ed3e6aed-e63e-438e-997e-503b599e0ddc',
        '61/5/59f9e614-d7dd-456a-adcf-48ed97ee2615',
        '71/9/b9fae65c-8a97-4841-8158-e10447f71719',
        'f4/a/bbf8a239-9102-4093-a3f6-7c1b110cdf4a',
        'ec/9/5f4d9b86-94f0-43fb-a8b7-4e21991daec9',
    ]

    # Retrieve the streams from S3
    def member_files():
        for key in s3_keys:
            url = sign_url(key)
            yield (key, now, 0o600, ZIP_64, get_s3_stream(url))

    # Set the headers
    headers = [('Content-Type', 'application/zip'),
               ('Content-Disposition', 'attachment; filename="large_file.zip"')]
    start_response('200 OK', headers)

    # Stream the zipped content
    get_compressobj = lambda: zlib.compressobj(level=0)
    return stream_zip(member_files(), get_compressobj=get_compressobj)

def get_s3_stream(presigned_url):
    with requests.get(presigned_url, stream=True) as response:
        response.raise_for_status()
        for chunk in response.iter_content(chunk_size=8192):
            yield chunk

Please note I'm yielding 8k chunks here, and it's working very well so far.

Is there a way to get total size so users can now how long until the download is complete?

@Wamy-Dev

Is there a way to get total size so users can now how long until the download is complete?

If you're compressing the files, no, because the total size cannot in general be known until (just before) the very end of the generation of the zip file
If you're not compressing, then probably yes... Does https://github.com/uktrade/stream-zip/issues/21 have anything that helps?


Running with --dry-run causes segfault with my config

Check CONTRIBUTING guideline first and here is the list to help us investigate the problem.

fluentd 0.14.8
Running inside docker in a container which is built on top of this image (which is, itself, based on centos): https://github.com/fabric8io/docker-fluentd/blob/master/Dockerfile
My child image contains additional dependencies, including the fluentd-redshift plugin.
My configuration:

<match fluent.**>
    @type null
</match>
<source>
    @type tail
    path "#{ENV['INPUT_FILE_PATTERN']}"
    pos_file /var/log/rs-containers.log.pos
    time_format %Y-%m-%dT%H:%M:%S.%NZ
    tag foo
    format json
    read_from_head true
</source>
<match foo>
    @type redshift
   REDSHIFT CONFIG REDACTED
</match>

Note that this config w/ my image works (in docker) when not specifying --dry-run
Running with --dry-run causes a segfault; I invoke the image as follows:
docker run -e MY_ENV=blah -v /my/dir/no1:/my/dir/no1 -v /my/dir/no2:/my/dir/no2 -it my_image:my_tag --dry-run
The segfault is repeatable.
The following logs are written to STDOUT:
out.txt

From your logs, you use ruby 2.3.0-p0.
Could you update your ruby version to 2.3.3?

Same issue with 2.3.3:
out.txt

Will have to get back to you shortly.  As I mentioned, I'm running from a centos image, and it seems that the latest ruby in the centos repos and software collection is 2.3.0.  Might have to build ruby

If someone hit this problem with latest ruby and fluentd, reopen the issue.


Only one URL from this domain is giving Errno 104

[x] I've verified and I assure that I'm running youtube-dl 2016.11.04

Before submitting an issue make sure you have:

[x] At least skimmed through README and most notably FAQ and BUGS sections
[x] Searched the bugtracker for similar issues including closed ones

What is the purpose of your issue?

[x] Bug report (encountered problems with youtube-dl)
[ ] Site support request (request for adding support for a new site)
[ ] Feature request (request for a new functionality)
[ ] Question
[ ] Other

$ youtube-dl --verbose -g http://video.lptv.org/video/2365880559/
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--verbose', u'-g', u'http://video.lptv.org/video/2365880559/']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.11.04
[debug] Python version 2.7.9 - Linux-3.16.0-4-586-i686-with-debian-8.6
[debug] exe versions: avconv 11.8-6, avprobe 11.8-6, rtmpdump 2.4
[debug] Proxy map: {}
WARNING: Unable to download webpage: [Errno 104] Connection reset by peer
WARNING: Unable to download webpage: [Errno 104] Connection reset by peer
ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 694, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 357, in extract
    return self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/pbs.py", line 529, in _real_extract
    self._sort_formats(formats)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 924, in _sort_formats
    raise ExtractorError('No video formats found')
ExtractorError: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Description & Other Information
This output is reproduced every time for this specific URL. All other URLs at this domain have worked flawlessly. I was running an outdated version of youtube-dl when I first received the above output. After upgrading (using youtube-dl -U) I am still receiving the same output. Below is some other output to provide a little more detail about the OS version.
$ uname -a
Linux hostName 3.16.0-4-586 #1 Debian 3.16.36-1+deb8u1 (2016-09-03) i686 GNU/Linux

Works fine for me. Can you watch it in browser in the first place from this host?

Post the output of youtube-dl --verbose --dump-pages http://video.lptv.org/video/2365880559/.

The video works just fine in the browser for me. Well now, I was able to get the URL using youtube-dl -g but it returned WARNING: Unable to download webpage: [Errno 104] Connection reset by peer before returning the video URL. After that when I attempt to get the thumbnail, it is failing again (output below). Also, below that is the output you requested. I use youtube-dl often at this site and it has never happened before but maybe the host server doesn't like me accessing the same page in a short time span. But I'm still able to watch the video just after receiving that error. So I'm still not sure.
$ youtube-dl -v --get-thumbnail http://video.lptv.org/video/2365880559/
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'-v', u'--get-thumbnail', u'http://video.lptv.org/video/2365880559/']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.11.04
[debug] Python version 2.7.9 - Linux-3.16.0-4-586-i686-with-debian-8.6
[debug] exe versions: avconv 11.8-6, avprobe 11.8-6, rtmpdump 2.4
[debug] Proxy map: {}
WARNING: Unable to download webpage: [Errno 104] Connection reset by peer
WARNING: Unable to download webpage: [Errno 104] Connection reset by peer
ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 694, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 357, in extract
    return self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/pbs.py", line 529, in _real_extract
    self._sort_formats(formats)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 924, in _sort_formats
    raise ExtractorError('No video formats found')
ExtractorError: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

Output you requested
$ youtube-dl --verbose --dump-pages http://video.lptv.org/video/2365880559/
[debug] System config: []
[debug] User config: []
[debug] Command-line args: [u'--verbose', u'--dump-pages', u'http://video.lptv.org/video/2365880559/']
[debug] Encodings: locale UTF-8, fs UTF-8, out UTF-8, pref UTF-8
[debug] youtube-dl version 2016.11.04
[debug] Python version 2.7.9 - Linux-3.16.0-4-586-i686-with-debian-8.6
[debug] exe versions: avconv 11.8-6, avprobe 11.8-6, rtmpdump 2.4
[debug] Proxy map: {}
[pbs]<PHONE_NUMBER>: Downloading widget/partnerplayer page
WARNING: Unable to download webpage: [Errno 104] Connection reset by peer
[pbs]<PHONE_NUMBER>: Downloading portalplayer page
WARNING: Unable to download webpage: [Errno 104] Connection reset by peer
ERROR: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.
Traceback (most recent call last):
  File "/usr/local/bin/youtube-dl/youtube_dl/YoutubeDL.py", line 694, in extract_info
    ie_result = ie.extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 357, in extract
    return self._real_extract(url)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/pbs.py", line 529, in _real_extract
    self._sort_formats(formats)
  File "/usr/local/bin/youtube-dl/youtube_dl/extractor/common.py", line 924, in _sort_formats
    raise ExtractorError('No video formats found')
ExtractorError: No video formats found; please report this issue on https://yt-dl.org/bug . Make sure you are using the latest version; type  youtube-dl -U  to update. Be sure to call youtube-dl with the --verbose flag and include its complete output.

This looks like intermittent connectivity issue between you and http://player.pbs.org/. In any case it works fine for me from multiple locations.


GetPrerequisites: add path of visual studio 10 to 14 in order to find dumpbin.exe.

Thanks!  Applied as commit c79f88b0197eeb927100769ef5c4cef0275acf3a.  I also followed up with commit a1c402910c2010c0ec91d6325b4119ce0fe8379b to try to handle non-default install locations.


Improve Debug Logs

add regex to parse log rows. There are capturing groups for event name, line number, timestamp etc. We can use those to easily filter only specific events like USER_DEBUG and then display it in a better way. For example padding line numbers so that the debugs are better aligned.
Main thing this add is support for multiline debugs logs, for example when you JSON.serializePretty

@jake-kirkman can we get this merged so that JSON.serializePretty debug statement display more than one line?
This also has regex that will let you filter on other types of log statements down the road.


hardened kernel: test 6.6 & remove obsolete 4.14 from patches.json

Description of changes
The latter is dropped from our source-tree already and thus exposed nowhere, so there's no point in keeping it in here.
Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

Removing it from the JSON while the update script still finds it means that the update script tries to add it back every time.

Huh, good point.
4.14 shouldn't be maintained for too long anymore, but in the long-run this is something to be fixed in the hardened update-script.


config.json: Add alternate solution location pattern.

When https://github.com/exercism/configlet/pull/13 is merged, configlet will support alternate patterns for specifying the example solution filename.
This is done via the "solutionpattern" key.
The value "\\.meta/solutions/[^/]*\\.rb" ensures there is a .rb file in the .meta/solutions directory for a problem.
(double backslashes are required due to JSON escaping.)
It's not critical to check the exact name of the file, just that there is at least one ruby file which we can assume is a valid solution file. (Otherwise PR reviewers are being too slack.)
Merging this early should have no negative effects to the current version of configlet, but does risk (very small chance) there being other incompatible changes to configlet before this PR is merged.
Oh, and it requires synchronisation with the move of the example files.
(This might make it more appropriate that this PR be merged with that one. #607 )

Added the default pattern too, so this will still work before the solutions are moved.
But waiting for https://github.com/exercism/configlet/pull/13 to be merged first is a good idea.


tries caching lfs

should limit bandwith usage by actions. see https://github.com/actions/checkout/issues/165

addresses the bandwidth issue for issue #147 by caching git lfs. Pretty sure this still fetches and pulls all the git lfs files via git lfs pull as discussed here


feat(chart): add capsule-proxy 0.6.0 as optional dependency

Adds Capsule-Proxy as subchart. Another notable changes is, that resources are unset by default. This is the standard for any helm chart. All the other changes help to make the chart-testing work.

Thank you @oliverbaehler!
LGTM


Don't set address while serial is on

Setting memory address in configure_memory while serial is still on isn't needed and inadvertently puts 0x03 into the control register which gets reused and causes bad reads.
Removing it solves the issue.
Note for future: I assume the bug doesn't show for configure_eprom because the control register stays 0 through that part causing the write to be skipped.

Not relevant any longer


Bug - in Freeze Graph tool when trying to convert Graph.pb for Mobile Deployment

System information
Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 7
TensorFlow installed from (source or binary): binary
TensorFlow version (use command below): release 1.6
Python version: 3.5
Bazel version (if compiling from source): -
GCC/Compiler version (if compiling from source): - N/A
CUDA/cuDNN version: using dell laptop
GPU model and memory: N/A
Exact command to reproduce:  python freeze_graph.py  --input_binary=true --input_graph=c:/tmp/output_graph.pb  --input_checkpoint=c:/tmp/_retrain_checkpoint --output_graph=c:/tmp/frozen_graph.pb --output_node_names=output_node
I have trained Tesnsorflow to identify 2 classes(my own classes) of images which has resulted in output_graph.pb, _retrain_checkpoint.data-00000-of-00001, _retrain_checkpoint.index, _retrain_checkpoint.meta, checkpoint files being generated.
I want to convert this graph file to mobile version so I used below command as mentioned in tutorials..
python freeze_graph.py  --input_binary=true --input_graph=c:/tmp/output_graph.pb  --input_checkpoint=c:/tmp/_retrain_checkpoint --output_graph=c:/tmp/frozen_graph.pb --output_node_names=output_node
I get this error:
_saveables = self._ValidateAndSliceInputs(names_to_saveables)
File "C:\Users\svm\AppData\Local\Continuum\anaconda3\envs\tensorflow\lib
site-packages\tensorflow\python\training\saver.py", line 677, in ValidateAndSli
ceInputs variable)
TypeError: names_to_saveables must be a dict mapping string names to Tensors/Var
iables. Not a variable: Tensor("final_retrain_ops/biases/final_biases:0", shape=
(2,), dtype=float32)

Nagging Assignee @petewarden: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

I tried to freeze the retrained model from https://www.tensorflow.org/tutorials/image_retraining section "Training on Flowers" as per instruction given in https://www.tensorflow.org/mobile/prepare_models .. I still get the same error.. Request you to provide some info on how to get around this error

Any help?

Nagging Assignee @petewarden: It has been 22 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @petewarden: It has been 37 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @petewarden: It has been 52 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @petewarden: It has been 67 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @petewarden: It has been 82 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @petewarden: It has been 97 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @petewarden: It has been 112 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @petewarden: It has been 127 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @petewarden: It has been 142 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @petewarden: It has been 157 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @petewarden: It has been 172 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @petewarden: It has been 187 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Nagging Assignee @petewarden: It has been 202 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

Sorry for the slow response. The output of the retraining tutorial should already be frozen, so there's no need to freeze it again. The error is unclear though, it should be better, but closing this bug.


Implement DataFrameWriterV2

Description
The initial DataFrameWriter is created but there is also another way to write data via DataFrameWriterV2. This method has a slightly different implementation and leverages a different proto command message.
I think the methods should mirror the ones found on the Spark API guide, and a new method should be added onto the DataFrame for writeTo.

This looks fairly straightforward, I can pick it up.

Sounds good! Assigned to you. Thanks!

Hi @hntd187 , are you currently working on this issue?

@sjrusso8 chatted with @hntd187 and I think I can get this out quickly enough (maybe this weekend?). I want to start contributing to spark-connect, and this seems like perhaps a good way to do it. Unless you have better ideas!

Closed #30


Add translation helper API

Add an API to help with automatic language translation of markdown documents.

Added to 0.34.0 https://github.com/vsch/flexmark-java/wiki/Translation-Helper-API

Repo updated. Maven release in progress.