Status of chunking and resuming uploads

I've been searching through the documentation, but I didn't find anything about chunking and resuming of uploads. I'm asking, because I plan to use shrine with the fineuploader. There is a pretty nice documentation, what the backend should be capable of in here: http://docs.fineuploader.com/endpoint_handlers/traditional.html
So my question is, what should I do with UUID from the fineuploader? Does Shrine support chunking? How about resuming of uploads?
Thank you in advance for your answer :-)

Thank you for asking this, now I will not close this issue like Paperclip did, until we reach a satisfying result 😃
You're right, Shrine at the moment doesn't support chunked or resumable uploads. This is something that I thought about in the past, however, my conclusion was it makes much more sense to direct upload to an external service which does support these things.
Amazon S3 supports multipart uploads, and with that resumable uploads as well. From what I can see, FineUploader has support for uploading files directly to S3 and, unlike jQuery-File-Upload, it can do so in chunks by utilizing S3's multipart API. You can see Direct Uploads to S3 on how to setup direct S3 uploads with Shrine, along with some client-side tips. Would that be feasible for you?
In general I'm not closed to the idea of implementing chunked/resumable uploads in Shrine, but in that case instead of implementing a library-specific protocol like FineUploader imposes, it would be much better to implement TUS. Fortunately, there is already rubytus gem which gives you an app that implements the TUS protocol, storing files on the filesystem, and the author says that his company has been using it in production since 2013 (https://github.com/tus/tus.io/issues/28#issuecomment-149789948).
Let me know if any of these options would work for you.

Wow! Such an immediate response. Thank you :-)
Using S3 or Azure is not an option for this project, because of the data protection law, the project needs to be hosted in a specific country.
As far as I understand chunked uploads are specified in HTTP 1.1, whereas resumable uploads are not specified. Unfortunately FineUploader seems to use a library-specific protocol for chunked and resumable uploads.
Probably I need chunking, because of very big files. Then resuming is also super useful. I don't know yet, what I going to do.

I completely understand if you cannot use S3 or Azure; the goal of Shrine is to work for everyone, so we'll figure something out 😃
There is something called "chunked transfer encoding" which is part of HTTP 1.1, however, while it does allow sending data in chunks, I'm almost 100% sure that it doesn't let you specify the order. This means that you cannot upload chunks in parallel, or retry a chunk.
So, to give you more insight about RubyTus, it's a Rack application that accepts file uploads with the TUS protocol, and the end result is a file on the filesystem. That means that you can define Shrine's :cache directory to be the one where RubyTus saves the files, and then on client-side you can just add the location into Shrine's JSON representation of an attachment, the same way you do with direct S3 uploads. So if we can upload files to RubyTus, it should be good, right?
I think FineUploader wouldn't work in that case, as its protocol is pretty much sealed. Maybe there are apps which implement uploads with this protocol, just like RubyTus does for TUS. But instead of FineUploader you could just use a JS file upload library which supports TUS. The official one is tus-js-client.
How does that sound?

Sorry for not responding already. The weather was great and my motorbike smiled at me :-)
I really like the idea to use the well specified TUS protocol and I would love to go that way, if it doesn't consume too much time. I've been doing some research and I didn't find a popular uploading framework, which supports TUS, even though many of them have requests for TUS in their issue trackers (Plupload, FineUploader)
So I see two possibilities, which could work:

We could work together on that. We implement the support for TUS in Shrine and we implement in one of the JS uploading frameworks support for TUS. I think this would be a benefit for Shrine and makes it more popular, but I think it's more time consuming than the other possibility below. As I see we live in the same timezone and I've some time for that on the upcoming weekends. It would be fun for me doing something like that. The TUS protocol is already implemented in JavaScript and it doesn't have dependencies. I think it would be quite easy to implement it into FineUploader.

I implement the FineUploader protocol as a plugin in Shrine. I think it's not impossible and I estimate 20 hours of time for doing that. I'm so into FineUploader, because it's already in use in another project.

I've implemented the uploading today in my project with Shrine and FineUploader, but of course without chunking and resuming.
What do you think of all that?

Hey there, Fine Uploader developer here. The case where TUS support has been discussed in Fine Uploader can be found at FineUploader/fine-uploader#1620. In there, you'll also see historical information that explains why FU did not use TUS (TL;DR - chunking in FU came before TUS).
While it would be great to add optional TUS support to FU, I think the easier option, in the context of this case, is to add support for Fine Uploader chunked requests to shrine. Then again, I'm not familiar with shrine, so I could be way off-base with that assertion. Either way, I'm available to provide support and advice on the Fine Uploader side of things.

@openscript If we decide to go for the 1st option, there is actually nothing that needs to be done on the Ruby side, because as I mentioned there is already a production-ready implementation of TUS, Rubytus. I think there would we very little benefit in writing a new TUS implementation as a Shrine plugin; you could just do direct uploads like to any other service. IMO it's much better to use an existing well-tested solution, and it can easily be hooked up with Shrine.
If we go with the 2nd option (@rnicholus thank you very much for your input), I think it would be also great to implement a generic server (like Rubytus), something like "fineupload-ruby-server". It would just be a Rack application (I personally prefer Roda, but Rubytus chose Goliath), which could then be either mounted in your router or run separately. I think it would cool to have that, as FineUploader looks like a really advanced JS file upload library.
Let me know if you would like to take a stab at it. Otherwise I could try taking a look into it the next week. Or we could work on it together; it's been so long since I last pair programmed 😃
As this is technically not related to Shrine, I will close this issue. But feel free to continue the discussion here, or we can also switch to email.

My Ruby skills are pretty non-existent rusty. But if any FU-related questions come up, please do let me know. If this is a generic ruby server, perhaps this can be developed as a repo in the Fine Uploader org.

@rnicholus Thanks, if we decide to build a ruby server for FineUploader, I also think that the "FineUploader" organization would be a great place for it.
I think in the long run FineUploader would greatly benefit from having support for TUS, because it's a standalone and fully specified protocol, and really many people collaborated to bring it to 1.0, with contributions from companies like Vimeo. I think it's great to agree on a generic protocol and make servers for it, and then have different JS libraries which support that same protocol.
@openscript However, FineUploader's protocol is also nicely specified, and many server implementations exist as well. And if FineUploader has significant advantages over libraries like tus-js-client, then it might be worthwhile to implement a Ruby server for it. Even though that also sounds like a very decent amount of work, for me personally it would be easier, because my JavaScript skills are not so hot, but my Ruby skills are.
But, there is actually a 3rd option that comes to mind. What if we created a Rack middleware that sits in front of Rubytus, and translates the FineUploader protocol into TUS? That way we could leverage Rubytus, as I think it already solves a lot of problems which are protocol-agnostic, without having to change anything from FineUploader's side.
@openscript From these three options I'm now most inclined to start with the 3rd, as that seems like it would be the least amount of work, and we get to reuse existing generic components which is awesome. This way our "fineuploader server" would automatically benefit from any improvements that land on Rubytus. What do you think?

Actually, I'm not so sure that the 3rd option would require less work than others, because it requires understanding FineUploader protocol, TUS protocol, and Rubytus itself. But I think it would be the more useful than implementing a standalone FineUploader server.
It's also worth noting that Rubytus doesn't seem to fully support TUS 1.0 yet (https://github.com/picocandy/rubytus/issues/2), so perhaps it wouldn't automatically work with tus-js-client.
There is also another option, I think, and that is to use an existing FineUploader server written in another language. fineuploader-go-server looks really good and would probably be the best bet, because from what I understood you don't need to have Go installed on the server to run a Go program. It might sound strange to use an app written in another language, but from the client side you don't actually care in what language an app is written in, you just care that it accepts requests and returns responses. And you shouldn't need to modify anything, just run it on some port like any other app.
To summarize, I think we have five options here:

Instead of FineUploader use tus-js-client with Rubytus
Add TUS support to FineUploader and use Rubytus (probably most useful in the long run)
Implement a standalone FineUploader-compliant Ruby server
Implement a Rack middleware for Rubytus which translates FineUploader requests/responses into TUS (although I don't know if this is even possible)
Use an existing FineUploader-compliant server like fineuploader-go-server

@openscript I actually don't think I will have time to help with 3rd or 4th option, so it's up to you to choose whatever is best for you.

I read through the whole issue again and tried to come to a conclusion, where to go from here. I've been working on several Rails projects since 2011 and handling uploaded files was always a pain. In my opinion there was never a rock solid and flexible solution for doing that. I've tried many combinations of Paperclip, Carrierwave, PluploadJS, jQuery File Upload or FineUploader. There was always something I wasn't happy about or something which needed some wizardry and tinkering. It would be great to have a strong, flexible, foolproof solution for uploading files to Ruby apps.
During today I made for all options a little diagram, so it's easier to get an overview. I wrote down the pros and cons as well:
Option 1

Pros:

No changes in FineUploader
In my opinion, this solution takes the least time

Cons:

FineUploader doesn't get TUS support
The RubyFineUploader endpoint has only a single purpose.
There still won't be a strong file uploading framework with TUS support
There are quite many components, which have to work together

Option 2

Pros:

FineUploader gets TUS support.
There will be a strong file uploading framework with TUS support.
Not so many components. No new component. Building on existing projects and components.

Cons:

Integrating TUS into FineUploader can be time consuming.

Option 3

Pros:

No changes in FineUploader

Cons:

FineUploader doesn't get TUS support
The FUToTUS endpoint has only a single purpose.
There still won't be a strong file uploading framework with TUS support
There are quite many components, which have to work together.
A lot of knowledge needed for the FUToTUS component: Ruby, FineUploader protocol, TUS, ..

Option 4

Pros:

Almost nothing to implement (maybe extending RubyTUS a bit, but that needs to be done for the other options too) to get resumable uploads working. Just linking existing components together.

Cons:

FineUploader doesn't get TUS support
There still won't be a strong file uploading framework with TUS support
A lot of features missing, which are already implemented (AND tested) in FineUploader like: Client side image resize, Folder drop, ..

Option 5

Pros:

No changes in FineUploader
In my opinion, this solution doesn't consume a lot of time too..

Cons:

FineUploader doesn't get TUS support
Make the whole thing more complex by bringing another technology into the game
There still won't be a strong file uploading framework with TUS support
There are quite many components, which have to work together

I want to go for option 2. I've already forked FineUploader and started integrating TUS. I've set up and got the TUS server written in go runningm so Ihave something to run FineUploader with TUS against. I think with some support from @rnicholus this is getting on track soon!
It would be great to have a sample project with Shrine and RubyTUS. I haven't been digging to deep into RubyTUS, but I've seen too, that the last commit is a year ago and TUS 1.0 is not complete. Probably we need to get RubyTUS to TUS 1.0, but I think that's also not so hard. @janko-m do you think you have the time to set that example project up? :-)
What do you think of all that? Do you think it's possible to get the option 2 implemented until the 21. August?

@openscript You've spent quite a bit of time outlining all of the options, very impressive. Since the change here will be to Fine Uploader (option 2), can you outline your proposed changes to Fine Uploader in FineUploader/fine-uploader#1620? Perhaps I can offer some advice or locate a potential issue before you get too deep into implementing your solution.
The chunking logic in Fine Uploader is quite complex, mostly due to the concurrent chunking feature I added a couple years ago. This feature allows bandwidth to be maximized when uploading a single file. Essentially, it breaks the file up into multiple chunks and sends as many of the chunks as possible (while respecting the maxConnections option) at once. This was quite difficult to implement, and I realized why other libraries punted on this feature in the past, but the speed gains are notable.

@openscript Wow, so impressed with these diagrams and the pros & cons sections. It's great that you're going for adding TUS support to FineUploader. @rnicholus It's awesome that you've implemented concurrent chunking feature into FineUploader, hopefully that will work the same way with TUS.
@openscript Yes, I will gladly set up an example project that integrates Shrine and Rubytus. I will first use tus-js-client and check whether Rubytus works with it, and try to make necessary PRs if it doesn't. And later if you manage to get TUS into FineUploader, we'll switch tus-js-client with FineUploader.
I would also like to get an opinion from @MarkMurphy, as I was tracking his involvement around resumable uploads on Paperclip, and he seemed very knowledgeable about the topic. Mark, if you're not too busy, how did you end up implementing resumable uploads? Did Rubytus work well for you? If yes, what JS library did you use with it?

@janko-m Thanks, and yes I did help write some of the TUS protocol. I built a custom solution for a Ruby on Rails api. Some of it was inspired by code from Rubytus. I set it up so that it would easily couple with Paperclip. The api is currently only consumed by an iOS application so I can't speak for any client side libraries.
I don't mind sharing some of my api code.
Here's the gist:
https://gist.github.com/MarkMurphy/76dae9307cb67d56951e13a63df99b19
I'd definitely like to simplify it more and move the upload details into a database.

@MarkMurphy Thank you very much for sharing this!

I've added a comment about the progress with the TUS implementation in FineUploader:
https://github.com/FineUploader/fine-uploader/issues/1620#issuecomment-238994943

@openscript Just to report the status of the Shrine TUS demo.
So, Rubytus at the moment doesn't work with tus-js-client, I'm trying to fix it now. It is failing on the CORS preflight request, and I noticed that Rubytus was using old headers, so I tried copying over CORS headers from tusd, but for some reason it's still failing, I need to figure out why.
In the meanwhile I tried using tusd (a Go server), since it's the official one, and it's mature and up-to-date. And I managed to get things working. To really test resumability I tried using S3 backend for tusd (so that it's really uploading remotely), and it worked as well. However, the resumability didn't work, because tus-js-client only seems the remember the file once it's fully uploaded (it makes a request to tusd with the wrong ID when trying to resume). That's probably just a bug in tus-js-client, but that is irrelevant if we're going to use FineUploader.
So, once the TUS server finishes uploading, it returns the URL to the file, and I use shrine-url to store that URL as a cached file, which can then be promoted to any Shrine storage. So, the only part that's not working properly is uploading itself, so the part that's unrelated to Shrine.

@openscript So, I managed to get Rubytus working with tus-js-client, and now it works locally both creating and resuming (although there might be an edge case that I've missed). I made the following changes:

**Rubytus changes**
```diff
commit c8c83736994124db3c337e22b4195591c15de64a
Author: Janko Marohnić
Date:   Sat Aug 13 02:38:06 2016 +0800
Get things working

diff --git a/.ruby-version b/.ruby-version
deleted file mode 100644
index 68b3a4c..0000000
--- a/.ruby-version
+++ /dev/null
@@ -1 +0,0 @@
-1.9.3-p551
diff --git a/lib/rubytus/constants.rb b/lib/rubytus/constants.rb
index d93af20..49f63cd 100644
--- a/lib/rubytus/constants.rb
+++ b/lib/rubytus/constants.rb
@@ -16,7 +16,7 @@ module Rubytus
DEFAULT_MAX_SIZE       =<PHONE_NUMBER>
 SUPPORTED_VERSIONS     = ['1.0.0']

SUPPORTED_EXTENSIONS   = ['file-creation']

SUPPORTED_EXTENSIONS   = ['creation']
STATUS_OK                  = 200
STATUS_CREATED             = 201
diff --git a/lib/rubytus/helpers.rb b/lib/rubytus/helpers.rb
index 21dc232..9b33ffa 100644
--- a/lib/rubytus/helpers.rb
+++ b/lib/rubytus/helpers.rb
@@ -1,6 +1,7 @@
require 'rubytus/constants'
require 'rubytus/common'
require 'rubytus/error'
+require 'base64'

module Rubytus
module Helpers
@@ -12,7 +13,7 @@ module Rubytus
   env['api.headers'].merge!(handle_cors(request, headers))

 if request.collection? || request.resource?

 if (request.collection? || request.resource?) && !(request.options? || request.get?)
   validates_supported_version(headers["Tus-Resumable"])
 end

@@ -27,7 +28,7 @@ module Rubytus
 def validates_offset(req_offset, info_offset)
   if req_offset > info_offset

   error!(STATUS_FORBIDDEN, "Offset: #{req_offset} exceeds current offset: #{info_offset}")

   error!(STATUS_FORBIDDEN, "Upload-Offset: #{req_offset} exceeds current offset: #{info_offset}")
 end

end

@@ -71,18 +72,26 @@ module Rubytus
 def parse_metadata(metadata)
   return if metadata.nil?

 arr = metadata.split(' ')

 pairs = metadata.split(",").map { |string| string.split(" ") }

 if (arr.length % 2 == 1)

 if pairs.any? { |pair| pair.size != 2 }
   error!(STATUS_BAD_REQUEST, "Metadata must be a key-value pair")
 end

 Hash[*arr].inject({}) do |h, (k, v)|

 Hash[pairs].inject({}) do |h, (k, v)|
   h[k] = Base64.decode64(v)
   h
 end

end

def serialize_metadata(metadata)

 encoded_list = metadata.map do |key, value|

   "#{key} #{Base64.encode64(value)}"

 end

 encoded_list.join(",")

end

def handle_cors(request, headers)
origin = headers['Origin']

@@ -92,11 +101,11 @@ module Rubytus
cors_headers['Access-Control-Allow-Origin'] = origin
   if request.options?

   cors_headers['Access-Control-Allow-Methods']  = "POST, HEAD, PATCH, OPTIONS"

   cors_headers['Access-Control-Allow-Headers']  = "Origin, X-Requested-With, Content-Type, Entity-Length, Offset, TUS-Resumable"

   cors_headers['Access-Control-Allow-Methods']  = "POST, GET, HEAD, PATCH, DELETE, OPTIONS"

   cors_headers['Access-Control-Allow-Headers']  = "Origin, X-Requested-With, Content-Type, Upload-Length, Upload-Offset, Tus-Resumable, Upload-Metadata"
   cors_headers['Access-Control-Max-Age']        = "86400"
 else

   cors_headers['Access-Control-Expose-Headers'] = "Offset, Location, Entity-Length, TUS-Version, TUS-Resumable, TUS-Max-Size, TUS-Extension"

   cors_headers['Access-Control-Expose-Headers'] = "Upload-Offset, Location, Upload-Length, Tus-Version, Tus-Resumable, Tus-Max-Size, Tus-Extension, Upload-Metadata"
 end

 cors_headers

@@ -117,7 +126,7 @@ module Rubytus
env['api.uid']           = uid
env['api.entity_length'] = request.entity_length
env['api.resource_url']  = request.resource_url(uid)

   env['api.metadata']      = parse_metadata(headers['Metadata'])

   env['api.metadata']      = parse_metadata(headers['Upload-Metadata'])
 end

end

@@ -134,6 +143,10 @@ module Rubytus
env['api.action']  = :patch
env['api.buffers'] = ''
env['api.offset']  = request.offset
+

   env['api.headers'].merge!(

     'Upload-Offset' => (request.offset + request.content_length).to_s

   )
 end

 if request.get?

diff --git a/lib/rubytus/info.rb b/lib/rubytus/info.rb
index 7e41e0d..ad3cbd7 100644
--- a/lib/rubytus/info.rb
+++ b/lib/rubytus/info.rb
@@ -3,25 +3,33 @@ require 'json'
module Rubytus
class Info < Hash
def initialize(args = {})

 self['Offset']       = args[:offset]        || 0

 self['EntityLength'] = args[:entity_length] || 0

 self['Meta']         = args[:meta]          || nil

 self['Upload-Offset']   = args[:offset]        || 0

 self['Upload-Length']   = args[:entity_length] || 0

 self['Upload-Metadata'] = args[:meta]          || nil

end
def offset=(value)

 self['Offset'] = value.to_i

 self['Upload-Offset'] = value.to_i

end
def offset

 self['Offset']

 self['Upload-Offset']

end
def entity_length=(value)

 self['EntityLength'] = value.to_i

 self['Upload-Length'] = value.to_i

end
def entity_length

 self['EntityLength']

 self['Upload-Length']

end

def metadata=(value)

 self['Upload-Metadata'] = value

end

def metadata

 self['Upload-Metadata']

end
def remaining_length
diff --git a/lib/rubytus/middlewares/storage_barrier.rb b/lib/rubytus/middlewares/storage_barrier.rb
index 786d3c6..7c36537 100644
--- a/lib/rubytus/middlewares/storage_barrier.rb
+++ b/lib/rubytus/middlewares/storage_barrier.rb
@@ -1,9 +1,12 @@
+require 'rubytus/constants'
+require 'rubytus/helpers'
require 'rubytus/error'

module Rubytus
module Middlewares
class StorageBarrier
include Rubytus::Constants

 include Rubytus::Helpers
 include Goliath::Rack::AsyncMiddleware

 def post_process(env, status, headers, body)

@@ -16,7 +19,7 @@ module Rubytus
when :create
status = STATUS_CREATED
headers['Location'] = env['api.resource_url']

       storage.create_file(env['api.uid'], env['api.entity_length'])

       storage.create_file(env['api.uid'], env['api.entity_length'], env['api.metadata'])

     when :head
       info = storage.read_info(env['api.uid'])

@@ -24,11 +27,28 @@ module Rubytus
if info.nil?
status = STATUS_NOT_FOUND
else

         headers['Offset'] = info.offset.to_s

         headers['Upload-Offset']   = info.offset.to_s

         headers['Upload-Length']   = info.entity_length.to_s

         headers['Upload-Metadata'] = serialize_metadata(info.metadata) if info.metadata
       end

     when :get

       body = storage.read_file(env['api.uid'])

       path = storage.file_path(env['api.uid'])

       info = storage.read_info(env['api.uid'])

       if metadata = info.metadata

         headers['Content-Disposition'] = "attachment; filename=\"#{metadata['filename']}\"" if metadata['filename']

         headers['Content-Type'] = metadata['content_type'] if metadata['content_type']

       end

       file = ::Rack::File.new(File.dirname(path))

       file.path = path

       result = file.serving(env)

       status  = result[0]

       headers = result[1].merge(headers)

       body    = result[2]
     end
   rescue PermissionError => e
     raise Goliath::Validation::Error.new(500, e.message)

diff --git a/lib/rubytus/request.rb b/lib/rubytus/request.rb
index 0b69abe..c8d9f1f 100644
--- a/lib/rubytus/request.rb
+++ b/lib/rubytus/request.rb
@@ -43,11 +43,11 @@ module Rubytus
end
 def entity_length

 fetch_positive_header('HTTP_ENTITY_LENGTH')

 fetch_positive_header('HTTP_UPLOAD_LENGTH')

end
def offset

 fetch_positive_header('HTTP_OFFSET')

 fetch_positive_header('HTTP_UPLOAD_OFFSET')

end
def base_path
@@ -88,8 +88,8 @@ module Rubytus
error!(STATUS_BAD_REQUEST, "#{header_orig} header must not be empty")
end

 if (header_name == 'HTTP_ENTITY_LENGTH') && (value < 0)

   error!(STATUS_BAD_REQUEST, "Invalid Entity-Length: #{value}. It should non-negative integer or string 'streaming'")

 if (header_name == 'HTTP_UPLOAD_LENGTH') && (value < 0)

   error!(STATUS_BAD_REQUEST, "Invalid Upload-Length: #{value}. It should non-negative integer or string 'streaming'")
 end

 value

diff --git a/lib/rubytus/storage.rb b/lib/rubytus/storage.rb
index c00a3c1..a0fe2f5 100644
--- a/lib/rubytus/storage.rb
+++ b/lib/rubytus/storage.rb
@@ -48,8 +48,9 @@ module Rubytus
def create_file(uid, entity_length, metadata = {})
fpath = file_path(uid)
ipath = info_path(uid)

 info  = Rubytus::Info.new(metadata)

 info  = Rubytus::Info.new
 info.entity_length = entity_length

 info.metadata = metadata if metadata

 begin
   File.open(fpath, 'w') {}

@@ -94,7 +95,7 @@ module Rubytus
   begin
     data = File.open(ipath, 'r') { |f| f.read }

   JSON.parse(data, :object_class => Rubytus::Info)

   parse_info(data)
 rescue SystemCallError => e
   raise(PermissionError, e.message) if e.class.name.start_with?('Errno::')
 end

@@ -113,5 +114,13 @@ module Rubytus
raise(PermissionError, e.message) if e.class.name.start_with?('Errno::')
end
end
+

private

def parse_info(data)

 info = Rubytus::Info.new

 info.replace(JSON.parse(data))

 info

end
end
end
diff --git a/test/rubytus/test_command.rb b/test/rubytus/test_command.rb
index 368da77..48f668a 100644
--- a/test/rubytus/test_command.rb
+++ b/test/rubytus/test_command.rb
@@ -204,7 +204,7 @@ class TestRubytusCommand < MiniTest::Test
:path => "/uploads/#{uid}",
:body => 'abc',
:head => protocol_header.merge({

   'Offset'        => '0',

   'Upload-Offset' => '0',
   'Entity-Length' => '3',
   'Content-Type'  => 'plain/text'
 })

@@ -230,7 +230,7 @@ class TestRubytusCommand < MiniTest::Test
:path => "/uploads/#{ruid}",
:body => 'abc',
:head => protocol_header.merge({

   'Offset'        => '0',

   'Upload-Offset' => '0',
   'Entity-Length' => '3',
   'Content-Type'  => 'application/offset+octet-stream'
 })

@@ -256,7 +256,7 @@ class TestRubytusCommand < MiniTest::Test
:path => "/uploads/#{ruid}",
:body => 'abc',
:head => protocol_header.merge({

   'Offset'        => '3',

   'Upload-Offset' => '3',
   'Entity-Length' => '3',
   'Content-Type'  => 'application/offset+octet-stream'
 })

@@ -281,8 +281,8 @@ class TestRubytusCommand < MiniTest::Test
:path => "/uploads/#{ruid}",
:body => 'abcdef',
:head => protocol_header.merge({

   'Offset'        => '0',

   'Entity-Length' => '6',

   'Upload-Offset' => '0',

   'Upload-Length' => '6',
   'Content-Type'  => 'application/offset+octet-stream'
 })

}
@@ -300,8 +300,8 @@ class TestRubytusCommand < MiniTest::Test
:path => "/uploads/#{uid}",
:body => 'abc',
:head => protocol_header.merge({

   'Offset'        => '0',

   'Entity-Length' => '3',

   'Upload-Offset' => '0',

   'Upload-Length' => '3',
   'Content-Type'  => 'application/offset+octet-stream'
 })

}
diff --git a/test/rubytus/test_storage.rb b/test/rubytus/test_storage.rb
index b36fbee..7fe0584 100644
--- a/test/rubytus/test_storage.rb
+++ b/test/rubytus/test_storage.rb
@@ -48,12 +48,12 @@ class TestStorage < MiniTest::Test
def test_read_info
File.open(@storage.info_path(@uid), 'w') do |f|

 f.write('{"Offset":100,"EntityLength":500,"Meta":null}')

 f.write('{"Upload-Offset":100,"Upload-Length":500,"Meta":null}')

end
info = @storage.read_info(@uid)
assert_kind_of Hash, info

assert_equal 100, info['Offset']

assert_equal 100, info['Upload-Offset']
end

def test_read_info_failure
@@ -63,19 +63,19 @@ class TestStorage < MiniTest::Test
def test_update_info
File.open(@storage.info_path(@uid), 'w') do |f|

 f.write('{"Offset":100,"EntityLength":500,"Meta":null}')

 f.write('{"Upload-Offset":100,"Upload-Length":500,"Meta":null}')

end
@storage.update_info(@uid, 250)
info = @storage.read_info(@uid)
assert_kind_of Hash, info

assert_equal 250, info['Offset']

assert_equal 250, info['Upload-Offset']
end

def test_update_info_failure
storage = Rubytus::Storage.new(read_only_options)

stub(storage).read_info(@uid) { Rubytus::Info.new('Offset' => 100) }

stub(storage).read_info(@uid) { Rubytus::Info.new('Upload-Offset' => 100) }
assert_raises(Rubytus::PermissionError) { storage.update_info(@uid, 250) }
end

</details>

However, when attempting to make a pull request with organized commits, I realized that there are still some things in the specification that are not part of Rubytus, but I had troubles implementing it because Goliath is a complex web framework 😃

In Goliath you can return early by raising a validation error, but there currently isn't a way to set headers when raising an error. I [made a PR](https://github.com/postrank-labs/goliath/pull/327) to Goliath, which should get merged soon, but I'm not sure when a new version of Goliath will be released (the last version was released in June 2014), and we would need a version to put it in rubytus.gemspec.

However, I feel like using Goliath is really an overkill here, the asynchronicity makes it much more difficult to work with than e.g. Roda. And I feel that EventMachine only shines when you do slow IO, and here we're just writing to filesystem. Even after that, Rubytus is still missing some TUS extensions (notably `concatenation`, which I find really important because it enables parallel chunked uploads).

So, now that read the whole TUS 1.0 thoroughly, I decided that it makes much more sense to make my own implementation in Roda. That would make it much easier to maintain, since not many people are familiar with Goliath. And also current Rubytus code is pretty difficult to follow, since it's scattered across middlewares and subclasses. I feel like it would it would take similar amount of time to make Rubytus work fully by the specification and to make a Roda-based TUS 1.0 implementation. And it would bring me a lot of joy to do this, because I absolutely love Roda. 😃

It is done: https://github.com/janko-m/tus-ruby-server
I also created a demo app which integrates it with Shrine: https://github.com/janko-m/shrine-tus-demo

That is truly awesome @janko-m!


Update the continuous integration tests

This pull request update gimli.units continuous integration. The build/test step is now broken into separate build and test steps. The build step builds wheels and then uploads them as artifacts. The test step downloads these wheels and the runs the tests on them.

Pull Request Test Coverage Report for Build<PHONE_NUMBER>1
Details

0 of 0   changed or added relevant lines in 0 files are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+100.0%) to 100.0%

Totals

Change from base Build<PHONE_NUMBER>:
100.0%

Covered Lines:
6

Relevant Lines:
6

💛  - Coveralls


How to use the Mongo.command function

In the documentation for this database driver little is said about how exactly to use the Mongo.command function.
Reference: https://hexdocs.pm/mongodb/Mongo.html#command/3
I searched online for examples on how to use it but with no luck.
After some experimentation, If I wanted to do a find query then i would have:
          Mongo.command(:db_connection, %{
            find: "addresses",
            projection: %{}
          })

I wanted to use the Mongo.command function specifically to create a new user. How does the Mongo.command function work? Is there any way I can run the standard Mongo db function syntax?

create_user_command = %{
createUser: "username",
pwd: "some_password",
roles: []
}
Mongo.command(pid, create_user_command)
To view all available parameters for this command, please read the
following page https://docs.mongodb.com/manual/reference/command/createUser/#dbcmd.createUser

Thanks


Modified README.md to add Swift syntax highlighting

Hey! I found this via the trending Swift repos. This looks pretty interesting... thanks for creating this! I added Swift syntax highlighting to the README to make it a little easier to read.

Thanks for contribution in Swift syntax highlighting to README Wilson.

@KiranJasvanee Can you please make a swift 3 repo?


codespaces with python 3.10 and jupyter support added

tested both locally with vscode dev container and remote codespaces

Hey dantel, I see your commits yes, however I see also that the notebooks have your credentials. I need to merge only the .devcontainer and .github folders only

closed this one. created a new one easier to merge: https://github.com/pablomarin/GPT-Azure-Search-Engine/pull/46


fix(getHeadersArray) use resolveRef on $ref header FTI-2555

This resolves issues where createHar fails if there is a $ref is used for a header.

swagger2har does not currently support refs to external files.  This check
for ^http is seen in other places in code before doing resolveRef
On Fri, May 21, 2021, 5:30 PM kinman @.***> wrote:

@.**** commented on this pull request.
In src/swagger2har.js
https://github.com/Kong/swagger2har/pull/12#discussion_r637320038:

@@ -247,6 +247,13 @@ var getHeadersArray = function(swagger, path, method) {
for (var k in pathObj.parameters) {
var param = pathObj.parameters[k]
if (typeof param.in !== "undefined" && param.in.toLowerCase() === "header") {

   if (typeof param["$ref"] === "string") {

     // can't handle this yet

What functionality are we skipping / do we not support here?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/Kong/swagger2har/pull/12#pullrequestreview-666133352,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADZGUNCJR2RUK3PZW5HDIT3TO33MDANCNFSM45KAL4AQ
.


change unit

Is there a way to change unit from mg/dL to mmol/L?
At my country we usually use mmol/L and numbers are (or should be at best) around 5.5.
As the values are in these ranges, it messes up the graph.

@Fishbaits  The graph mess up is related to date issue. It should automatically pick up units data from your pump. If you can upload the CareLink*.json file from your documents folder I can look into it further.
The json file is readable text and personal data is removed if you save as snapshot from the Start Up Menu of the app.
There is a beta version you can try under https://github.com/paul1956/CareLink/files/9193835/CareLinkv<IP_ADDRESS>x64.zip which fixes the graph but looking at it I have some precision issues I need to address.

When I try to run this new version, it says so:

And then quits, no matter if I click OK or Cancel

@Fishbaits thanks. I will upload a trial fix tomorrow.
This information will help.

@Fishbaits It looks like the pump stores and reports everything in mg/dl and display software scales it. So all the data is there but off the top of the graph. It is going to take some time to scale everything. I was going to post something that works today but it would be in US units. I will work on fixing the units next week.

@Fishbaits
Fixed in CareLinkv<IP_ADDRESS>x64
Please open new issue if the issue persists


How to remove image from the server when deleted in rails

Has anyone managed to figure out how to delete a specific image from the server in rails?

@brookemckim  Doesn't work with the new version anymore :(


How to add Google material icons to the project?

I would like to add the following link in my html but I don't see a index.html file in the app. If I were to create one, where would I put that in the app?
<link href="https://fonts.googleapis.com/icon?family=Material+Icons"
      rel="stylesheet">

this is my main Styus file..
@import url(https://fonts.googleapis.com/css?family=Roboto:400,300,700,900);

h1
	font-family: 'Roboto', sans-serif
	font-weight: 700
	font-size 2rem

h2
	font-family: 'Roboto', sans-serif
	font-weight: 300
	font-size 1.4rem

h3
	font-family: 'Roboto', sans-serif
	font-weight: 400
	font-size 1.3rem

h4
	font-family: 'Roboto', sans-serif
	font-weight: 300
	font-size 1.1rem

Just create a style.css inside a component directory and import the script.

The material icons couldn't be imported using @import in CSS. Apparently, font awesome icons have same issue. I ended up downloading material icons in public folder to make it work.
Is there a reason why mantrajs doesn't allow access to index.html? It would be nice to be able to use at least the  portion for quick prototyping. I know we can use kadira:dochead to manipulate  but sometimes, it's just convenient to copy and paste links there.

We won't stop you from adding a HTML file and setup CSS. That's totally fine.

I am a little lost right now. Where would I put index.html in the mantra-sample-blog-app and ensure that the  is served correctly. Thanks.

Apparently, other people are having similar issues too. Here is an issue link from mantra kickstarter app.

@rnarayan you can put your <head></head> in configs.  Or if you want to use dochead, you can make a dochead.js and startup.js in the configs and initialize it in client/main

@sammkj is correct. Create a startup.html file in configs and add your head into that. Meteor will load it correctly.
You can use DocHead too (and helmet) as mentioned on the kickstarter app repo in the similar fashion.
If it's DocHead, it's a good idea to run them inside FlowRouter trigger to support server side rendering.

@rnarayan I'll try to add an example case here: https://github.com/mantrajs/mantra-sample-blog-app/issues/76

Thanks @sammkj and @arunoda. I will try your suggestion of startup.html in configs. I was planning on using DocHead but then the favicon issue worried me. I am using mantrajs to prototype a production app.


Expose Patricia Tree Module

patricia_tree module should be exposed as public for usage.

@apoelstra any objections to expose patricia_tree directly?

No objections


Problem with documents:infer service - returns 500 error

There is some problem with https://prod.json.systems/v1/documents:infer
You can try Submit default schema (Schema tab)
xxx=token
POST https://prod.json.systems/v1/documents:infer?key=xxx 500 ()
(index):1 Failed to load https://prod.json.systems/v1/documents:infer?key=xxx:
No 'Access-Control-Allow-Origin' header is present on the requested resource.
Origin 'https://jsonschema.net' is therefore not allowed access. The response had HTTP status code 500.

Thanks for taking the time to report this. Is it still a problem for you?  I can't recreate the issue, but that's not to say other people aren't experiencing it.

We can close it, looks like it was temp. server problem.
Thanks for repl Jack

@jsonsystems It seems that this issue has surfaced again.  I've tried a few different JSON samples, and I always get the same 500 error

@joeskeen Thank you for posting this issue.
I just tried this with the developer tools open and cache disabled and it works for me. Can you provide your browser details please? And when did this last work for you?

I have seen it the last tree days, eg just now: Thu Aug 27 22:26:12 CEST 2020
Logging out and back in did not help.
Tried Chrome and Safari on Mac OS.

@mdornseif Thanks for this information. Looks like it could be related to the JWT used for authentication (incorrect padding and because it worked when logged in with a guest session).

@mdornseif this should now be working, can you try www.jsonschema.net?


Share function

Share function not working properly - just empty link, in regular browser ok, can you fix it please? Android 14. Thx
broke recently, it was ok before

Can you share some screenshots

screenshot

After ota my phone - now it's ok, so close this issue


Hide/disable data json under videos

Is there a way to automatically disable or hide the json that shows below each embedded video? I.e. the dictionary that shows the "name" and "data" elements?

Hello @alicekemp,
Could you share the code you used?
The data shown below the video is returned by st_player(). I suspect you're printing it unintentionally like so:
from streamlit_player import st_player

data = st_player("url")
data  # This line will display the dictionary {"name": ..., "data": ...) below the video

Ah, I was defining the function as a variable and calling it - quick fix. Thanks for your help.


fixes docs after https://github.com/NixOS/nixpkgs/pull/60219

Things done
added missing xml-closing-tag in release notes
docs build on nixos, rest is still rebuilding.

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[x] NixOS
[ ] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Notify maintainers
cc @arianvp

Thanks so much. Didn't realise I broke the build :) lets get this merge

Superseded by https://github.com/NixOS/nixpkgs/commit/19a1e15501e9f47eb48a6bc39cd2e3085c319e86


Fix/dynamic menu

The implementation of the url and entity dynamic actions is only visible on shesha.
This is to make it visible in other projects. Wrapped the provider within the ShaApplicationProvider .

Related to this issue:
#2284


Enable Tide for dependency-watchdog

/kind enhancement
What this PR does / why we need it:
This PR enables TIDE integration for dependency-watchdog repository.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
/cc @oliver-goetz

Will raise another PR. Closing this one.


added PressDownArrow method to use in auto suggestion dropdown

Proposed changes
Added command to press down arrow. It will be really helpful during auto-suggestion dropdown without breaking the code reusability.
$(element).pressDownArrow().pressEnter()
Checklist

[x] Checkstyle and unit tests are passed locally with my changes by running gradlew check chrome_headless firefox_headless command
[ ] I have added tests that prove my fix is effective or that my feature works - not applicable
[x] I have added necessary documentation (if appropriate)

The code is good, but do we really need to add separate methods to all possible keys?
$.pressDownArrow() is the same as $.sendKeys(DOWN). The latter is even shorter.
P.S. Yes, I know we have methods $.pressEnter() and $.pressTab() - and probably it was a mistake for the same reason...

@asolntsev - You are correct but the sendkeys method is void. We normally use down arrow before press enter.
currently in our project we do something like
$.setValue("123").sendKeys(Keys.Down);
$.pressEnter()
From now on, we can use $(element)..setValue("123").pressDownArrow().pressEnter()

@amuthansakthivel This is a valid argument: current sendKeys method is void. Probably we could then introduce method $.press? Then it would be possible in one line:
$.press(DOWN).click();
$.press(TAB).should(disappear);

@amuthansakthivel This is a valid argument: current sendKeys method is void. Probably we could then introduce method $.press? Then it would be possible in one line:
$.press(DOWN).click();
$.press(TAB).should(disappear);

@asolntsev - Makes a whole lot of sense. Let me refactor.

@asolntsev - Do we need methods like press(Keys[] keysToPress) or press(Keys keyToPress, int numberOfTimesToPress)

It will be good to write at least one test to avoid regression in future.


Is it possible to increase the output image size?

How would I increase the output image size so that each bird is at a higher resolution? I recognize that changing the image size will not create as realistic of an output.

I tried changing both  __C.TEST.LR_IMSIZE = 64 and __C.TEST.HR_IMSIZE = 128 located in the StackGAN/misc/config.py file, but it results in the below error:
ubuntu@ip-Address:~/StackGAN$ sh demo/flowers_demo.sh
{
  doc_length : 201
  filenames : "Data/flowers/example_captions.t7"
  queries : "Data/flowers/example_captions.txt"
  net_txt : "models/text_encoder/lm_sje_flowers_c10_hybrid_0.00070_1_10_trainvalids.txt_iter16400.t7"
}
Successfully load sentences from:  Data/flowers/example_captions.t7
Total number of sentences: 5
num_embeddings: 5 (5, 1024)
lr_imsize:  128
Reading model parameters from ./models/flowers_model_130000.ckpt
WARNING:tensorflow:From demo/demo.py:72 in build_model.: all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Please use tf.global_variables instead.
Traceback (most recent call last):
  File "demo/demo.py", line 192, in <module>
    build_model(sess, embeddings.shape[-1], batch_size)
  File "demo/demo.py", line 73, in build_model
    saver.restore(sess, ckt_path)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1388, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 766, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 964, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1014, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1034, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [228,65536] rhs shape= [228,16384]
         [[Node: save/Assign_48 = Assign[T=DT_FLOAT, _class=["loc:@g_net/custom_fully_connected_1/Linear/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/cpu:0"](g_net/custom_fully_connected_1/Linear/Matrix, save/RestoreV2_48)]]

Caused by op u'save/Assign_48', defined at:
  File "demo/demo.py", line 192, in <module>
    build_model(sess, embeddings.shape[-1], batch_size)
  File "demo/demo.py", line 72, in build_model
    saver = tf.train.Saver(tf.all_variables())
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1000, in __init__
    self.build()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1030, in build
    restore_sequentially=self._restore_sequentially)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 624, in build
    restore_sequentially, reshape)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 373, in _AddRestoreOps
    assign_ops.append(saveable.restore(tensors, shapes))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 130, in restore
    self.op.get_shape().is_fully_defined())
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_state_ops.py", line 47, in assign
    use_locking=use_locking, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1128, in __init__
    self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Assign requires shapes of both tensors to match. lhs shape= [228,65536] rhs shape= [228,16384]
         [[Node: save/Assign_48 = Assign[T=DT_FLOAT, _class=["loc:@g_net/custom_fully_connected_1/Linear/Matrix"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/cpu:0"](g_net/custom_fully_connected_1/Linear/Matrix, save/RestoreV2_48)]]

ubuntu@ip-Address:~/StackGAN$

Thanks for the interests to the work. Currently, for our pre-trained model, the spatial size for stack-I image is 64 * 64 and 256 * 256 for the stack-II image. We have not tried higher resolution yet. If you have interests in generating higher resolution image, you can train the model by yourself by changing the yml file. Or even have more stacks, but it is helpful to make sure that the batch size you pick fits for the GPU memory.

It depends on the network structure you use. For the current one, it takes 2~3 days on a single TitanX Pascal. Or you can change the code to use multiple GPU for the training, it will helps reduce the training time.

@hanzhanggit  Could you upload your models to BaiDuYun? I can't download the models even use VPN.
Thank you very much.

@kli-nlpr I am working on creating a docker image for this repo so that folks can get up and running with this code quickly without having to follow the install instructions. As a part of this, I've written some scripts that download the pre-trained models. I had trouble curling from google drive, so I've downloaded the models and am hosting them on github here (see birds_model_164000.ckpt, flowers_model_130000.ckpt, and  birds_skip_thought_model_164000.ckpt). Hope this helps :)

@brannondorsey  Thank you very much

@hanzhanggit Also about the size of the image. In stage-1, the spatial size of image is 6464, and we hope the generator in stage-2 may receive missed information to generate the high-resolution images. I'm wandering if in stage-1 we generate image of size 256256 and in stage-2 the generator just plays as a super-resolution mechanism with the missed information, will it works as well. How do you think about it?Have you ever tested it before?


Android build crashes when model_complexity set to 0 in posetrackinggpu

Issue:
When building with model_complexity set to 0 (through variable select: 0) in pose_landmark_model_loader.pbtxt  the app crashes. It works fine when the same is default of 1. I'm simply running the sample app with only this change.
Platform: Android Mobile
Errors: No error while building, app apk starts and crashes.

Hi @Raghav997,
This can be an interesting finding. Would you let us know the complete steps followed to build the aar to understand the issue better. We suggest to fill the template to get maximum information to debug on the issue.

I just followed basic instructions given here.
[(https://google.github.io/mediapipe/getting_started/android.html)]
To replicate, just follow the instructions. You'll see the "1" model is working whereeas the lite "0" is crashing.

Any update on this?

Hi @Raghav997,
It looks like the two of one Tflite model files being missing from build file
"third_party/mediapipe/modules/pose_landmark/pose_landmark_lite.tflite" missing.
"third_party/mediapipe/modules/pose_landmark/pose_landmark_full.tflite" found.
Could you verify that "pose_landmark_lite" is actually missing from the BUILD file? If missing, you might need to add the dependency to pose_landmark_lite.tflite to the BUILD file.

In Ubuntu, I just downloaded the mediapipe git source ($ git clone https://github.com/google/mediapipe.git) and build the holistictrackinggpu android solution.
$ bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/holistictrackinggpu:holistictrackinggpu
and the holistictrackinggpu.apk (which was built in the bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/holistictrackinggpu/holistictrackinggpu.apk) is working well on my Android Phone.
Do you know how to set 0 for the value "model_complexity" for holistictrackinggpu Android ?
I just modified the mediapipe/mediapipe/modules/pose_landmark/pose_landmark_model_loader.pbtxt  as like below.
node {
calculator: "SwitchContainer"
input_side_packet: "SELECT:model_complexity"
output_side_packet: "PACKET:model_path"
options: {
[mediapipe.SwitchContainerOptions.ext] {
select: 1
and built again ($ bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/holistictrackinggpu:holistictrackinggpu)
but the holistictrackinggpu.apk is not working on the Android Phone.
I want to set 0 for the value "model_complexity" for holistictrackinggpu android.


Fix golint import path

I am a bot. Please reach out to @azillion if you have any issues, or just close the PR.

Coverage remained the same at 100.0% when pulling 6cdfe9da24feb334e1ad25d9040d527e1c23fa70 on golint-fixer:master into f32d4f033e2ce39fac8280ccdd1cdf21c9406fad on writescript:master.


Ghost Notifications

Hey ModMail devs.
 I've screenshotted the notification I get from windows, which upon clicking, takes me to a channel that I can't access (As if you got an invite link to a channel that you don't get permission to view). This has happened about 5 times today.
Strangest of all, I don't share a server with your bot, so I have no idea why I'm getting these notifications.
I can try and provide as much information as possible, but tbh this is one of the weirdest bugs I've ever seen.

I've been hearing similar reports of ghost notifications recently from other sources as well. It doesn't seem to be limited to modmail (or at least this specific bot), so I assume it's a bug of some sort on Discord's end. I'll see if I can track down/reproduce what's happening here!

Just started experiencing this issue a few hours ago. Not only I'm getting notifications from this bot, but also from users using the bot (some I don't even share a server with, although I could find them with the Ctrl+K menu).


Add DC for IBM MQ appliance

Why
Monitor the IBM MQ appliance.
What

Implemented a new type of otel host dc to monitor the IBM MQ appliance.
Added a README file for the host dc.

@liurui-1  Any comments here?

Let me check the code before merging.

It is better to bump up the version defined in build.gradle from 0.0.1 to maybe 0.1.0 because we have a new good plugin.

Updated the version of host dc to 0.1.0.


[1.1] Vendor response

Based on your README stating:

This is a beta release, so please report any issues you find. Reporting issues with the 1.1 patch would be
especially appreciated.

ROM v1.1 here.

Got a couple symbols of stray garbage in the vendor response upon purchase.
If this concerns you, let me know what additional debug info I can provide.
Strangely, only noticed this halfway through the first book. Unsure if it was present before.
In fact, I have played most of the first half of the first book on the 1.0 version of the ROM by mistake and later migrated the save (not a savestate) to 1.1, so that may be why. Could double-check by migrating back if you're planning to engage with investigating this.

I've seen this before. Looks like untranslated text.
https://github.com/RobertTheSable/fe3-translation/blob/master/project/text/shopping/35.txt
Unfortunately I can't actually read Japanese so I don't know what to replace it with.
It may be intended to be filler, I see this [_81]s pattern in a few other places.

I checked in the same shop with my 1.1 ROM and I haven't seen the text. So it's some kind of weird conditional text.

I have identified the missing text.

Apparently this text only happens when you buy an item in "supply mode" as Marth.

On further inspection, the text wasn't untranslated.
It looks like the "storage" table originally just had a list of item names, but either Quirino or VASM/RPGGuy (probably latter) added the "was sent to the storage text" to every single entry so that it fit the text box properly.
But they borked copying the substitution of the text, and so what should hove been a 00 81 in the rom became just 81. hence the weird garbage Japanese text.
Luckily, the vendor text works, so I can just repoint both this text and the text for the secret shop and it should be fine.

Fixed in 0.22.


[ADAP-939] [Bug] cannot reference permanent table from temporary table constraint when running incremental model with a foreign key

Is this a new bug in dbt-redshift?

[X] I believe this is a new bug in dbt-redshift
[X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior
From @amardatar on https://github.com/dbt-labs/dbt-core/issues/8022
Is this a new bug in dbt-core?

[X] I believe this is a new bug in dbt-core
[X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior
When running an incremental model with contract enforced and a foreign key, the model will fail on subsequent updates.
Expected Behavior
The update should succeed without an error.
Steps To Reproduce
First run a statement like:
create table if not exists numbers (n int not null primary key)

Then, given a schema.yml file:
version: 2

models:
  - name: test
    config:
      contract:
        enforced: true
      materialized: incremental
      on_schema_change: append_new_columns
      unique_key: n
    columns:
      - name: n
        data_type: integer
        constraints:
          - type: foreign_key
            expression: "numbers (n)"

And a SQL file:
select 1 as n

dbt will succeed on the first run, and subsequently fail with the error:
cannot reference permanent table from temporary table constraint

As per the error returned, the issue appears to be that while rendering a temporary table as part of the update, dbt will also render the foreign key constraint (which is not valid in the context of a temporary table). This leads to the above failure.
Relevant log output
No response
Environment
- OS: macOS Ventura 13.4.1
- Python: 3.11.4
- dbt: 1.5.2

Additional Context
While I've run into this issue using Redshift (and using the dbt-redshift adapter) I've put this bug in dbt-core as I believe from a bit of research that the same issue would occur if this were run in PostgreSQL (and possibly in other data warehouses, although it was harder to research the issue broadly).

@aranke verified a fix for Postgres in https://github.com/dbt-labs/dbt-core/pull/8768 and then created this backlog ticket for Redshift.

I think the fix for dbt-postgres was too large in scope, leading to other issues:

https://github.com/dbt-labs/dbt-core/issues/8857#issuecomment-1762261405

The original fix (https://github.com/dbt-labs/dbt-core/issues/8022) was modified slightly in https://github.com/dbt-labs/dbt-core/pull/8889 to still enforce the contract.

I have a customer who is interest in this being fixed. Anything we can do to move towards resolution? (e.g., @bdgeise's patch?)


Remove local copy of bech32 library.

Issue Number
#1333
Overview
This PR:

[x] Removes our local copy of the bech32 library.
[x] Includes the published bech32 repository as a git submodule.

@jonathanknowles we should also update the stack.yaml file to use the "remote" bech32 as an external dep instead of a local package. For this one at least, I think it makes sense to treat it as an external lib. What do you think?

@jonathanknowles we should also update the stack.yaml file to use the "remote" bech32 as an external dep instead of a local package. For this one at least, I think it makes sense to treat it as an external lib. What do you think?

I'd definitely be happy with adding it as an  extra-dep within stack.yaml.
If we do that however, why do we also need to have a git submodule?
I can understand the reason for adding a submodule later on, if we need to start working intensively with the bech32 code base, and if we need to frequently iterate with changes to cardano-wallet.
But for the moment, isn't it more simple to just add it as an ordinary dependency?
I was going to suggest just relying on stackage to select the latest version for us, but bech32 still isn't included in any LTS versions. (Though it is starting to appear in the nightly builds, starting from the end of last week.)
So we could add an entry in extra-deps for now, and then remove it later on after stackage includes it in an LTS version.
What do you think?

@KtorZ
I've updated the PR so that it:

[x] Specifies version 1.0.1 of the library from Hackage as an extra dependency within stack.yaml.
[x] Updates all the relevant nix machinery.

I've removed the git submodule for now. If we need it later on, we can add it back in again.

bors r+


Please have a new run!

This is great.

I will probably update all code bases & give it a new run this summer. I'm busy until then! :/

I am also looking forward to the new results.

I recommend using gcc (MinGW for windows) to compile angelscript. Because the standard string add-on of angelscript use std::string by default. And the performance of MSVC std::string is low (just allocate and copy, no reference counting and copy-on-write support).
I guess in most serious real cases where performance is a matter, almost all programmers will implement them own  string type by hand. For example, we can even eliminate the memory allocation and memory copy overhead of the String Factory by using the static data semantic (see: https://www.gamedev.net/topic/685672-is-it-safe-to-use-the-char-argument-of-string-factory-as-static-data/). So in real case the performance can be further improved.
The purpose of the test is to reflect the true level of the engine, So I think it is necessary to adjust the engine to its real world status.

Also, I suspect that register a line callback could dramatically slow down the engine. Use another timer thread which call ctx->Abort() on timeout instead.

I feel like there is a way this project can be architected such that these tests run on a VM and can be contributed by and perhaps even run by the community.  It would be work up front, but in the long run would both mitigate a lot of work that @r-lyeh has to do, and let us keep it updated.  Maybe we should brainstorm about ways to do that that aren't too difficult.

@rosshadden Like continuos integration? Whenever a library or test case is changed then run it again.

Yeah.

@asbai AFAIK, MinGW/GCC don't use reference counting and copy-on-write design for std::string. What they're using is a small-buffer optimization. Meaning, if your string is less than 16 characters, then it'll be stored in a local buffer and no heap allocation is performed. This makes the string type size be 8 bytes bigger than it would normally be and require some extra work (like when doing move semantics). But it pays off in the end because you avoid heap allocation for small strings and you never have to check for null buffer because there's always a buffer present. Also, why only 8 extra bytes when the local buffer is 16 bytes? It uses a union for the local buffer and capacity since the capacity is always known for the local buffer. Kinda like:
struct String {
    static constexpr size_t MINCAP = 16;

    char * m_Buffer;
    size_t m_Occupied;

    union {
        char    m_LocalBuffer[MINCAP];
        size_t  m_Capacity;
    };

    String() noexcept
        : m_Buffer(m_LocalBuffer), m_Occupied(0)
    {
        m_LocalBuffer[0] = '\0';
    }
};

If you're so hell bent on avoiding string copies where not necessary. I'd suggest investing some time in studying string/array views. Which were even added in the next C++17 standard.

Good ideas overall.

I've used CI a lot in most of my other repos and it's a valuable tool. Definitely @todo.
The VM idea sounds exciting and I am open to suggestions, what are the current options?
Current fibonacci test is somewhat lame, but very portable across all the implementations, plus it is handy to discard languages that lacks CC/tail optimization (also, bad for procedural stuff). Gotta add more computational/numeric tests. PRs welcome :)
Angelscript test should not use strings at all. If it does (like printing the fibonacci number), then it is my own mistake. Gotta review soon. No time right now :)

@iSLC No, you are wrong :-)
At least in my gcc 4.4.7 and  4.5.2, they all use reference count for std::basic_string, you can check it in bits/basic_string.h:
template<typename _CharT, typename _Traits, typename _Alloc>
class basic_string`
{
// Types:
public:
// ...
private:
struct _Rep_base
{
size_type		_M_length;
size_type		_M_capacity;
_Atomic_word	_M_refcount; // <---- here
};
// ...
In addition, I did not use the GCC string class, I use my own string class with reference counting and copy-on-write support.
PS: The algorithm you describe is very similar to what is used in VC++ STL, not GCC :-)

@r-lyeh Sorry, I can not found where the test script was stored, I guess you have some string performance test like this: https://codeplea.com/game-scripting-languages.
So which algorithms we are using now?

@asbai I'm talking about GCC 5.x, 6.x and future versions.

@iSLC Ok, so it's clearly that gcc 4.x is more benchmark friendly :-)
Especially for engines like AngelScript which has the ability to avoid the memory allocation and copy cost of the script / native calling conversion.
The memory allocation and copy cost of the native call is one of the biggest performance killer of dynamic languages.

@asbai, only fib.as is currently benchmarked. the native* and prime* tests are part of the original article you linked and are yet to be converted to the rest of languages before evaluation. such a difficult/boring task! :)
ps: the fib.as script is printing the result as I originally thought. this issue needs some global review before running the whole bench to be more fair at results.

@r-lyeh Ok, I think the ability of fast native calling conversion will not cause unfairness, because it is a feature of the language and the engine. Just like JIT is a feature of some engine.
But a line callback is unfair unless you have been registered this call back for every other engines. :-)

Aha thanks, will do in the next run :)


Use k8s.io/apimachinery yaml decoder in kubesource

Previously we used our own library for splitting multipart documents, but it had a bug (#18998). To avoid this in the future we now use the same decoder used by the k8s project.

In response to a cherrypick label: new pull request created: #19048


Elementum Burst broken

After last night Elementum Burst update, Elementum is not scrapping torrents anymore.
Full debug log here: http://paste.ubuntu.com/26184183/
Android 7 on Xiaomi MIBOX3 with SPMC v16 installed. Same thing happened on my KODI v17 instalaltion also.

Fixed with the latest v0.0.7


none-file based backends

Are there any plans to implement a backend that does not rely on file system backends?
If tang is running on a system with an ephemeral filesystem, some sort of cache on the FS may be fine, but the keys that currently reside in the db directory / sqlite database would be perfect candidates for an off-box object or KV storage.

I need to give a bit more thinking about the suggested use case.  I am currently occupied by other matter.  Hopefully, you can wait a little longer.


Clean Up ObservedFlowStatsMulti

Noticing some things that could be improved.
Specifically here: https://github.com/USGS-R/EflowStats/blob/master/R/ObservedStatsOtherMulti.R#L42 I don't think this is doing what it should.
And here: https://github.com/USGS-R/EflowStats/blob/master/R/ObservedStatsOtherMulti.R#L120 this should be a string match!
Would be worth sweeping the package for other places where sites are being treated as numerical types, this is totally subject to breaking because of the leading 0 issue.

Correct me if I am wrong, but from the code it looks like the drainage area filename must contain the character string "drain".
https://github.com/USGS-R/EflowStats/blob/master/R/ObservedStatsOtherMulti.R#L36
If so, it should be indicated in the function documentation.
This appears to be doing what its supposed to.

Fixed string match. Also added documentation indicating filename specifics.


Logo design proposal

Good day Sir @marcosgriselli , I am Tobaloidee and i am a Graphics Designer. I want to contribute to your nice project if you will permit me, regarding a logo design which i am going to do as a gift for free. I'll wait for your approval before I start designing Sir. I've noticed that it doesn't have a logo yet so i thought of helping out. Thanks and best regards!

Hi @Tobaloidee, absolutely! This project could definitely use a nice logo, I really appreciate it :).

Thanks for allowing me Sir..will start the design and show you here asap..

Sir, here's my design based from the feature of the project which is "swipeable tabbar"
favicon

logotypes

color options

let me know if you have some suggestions Sir so that i can edit.. :)

Sweet @Tobaloidee ! I really like it. I Feel the first logotype with Swipeable on the top and TabBarController on the bottom but with the same font and color fits better, otherwise it looks like the keyword is Swipeable and TabBarController is a subtitle.
I wonder if there's quality reduction here. Can you upload it to dropbox or drive and share the download link? I need the favIcon in 1024x1024 so I automatically generate them in all sizes necessary. That pink color looks good! 👍

Here Sir, pink version and adjusted logotypes...I can make a PR and upload the 1024x1024 SVG files for any scaling implementations if you like Sir.

Can I get the app icon only in 1024x1024? So I add it to the demo app. You can create a PR with the banner as SVG and add it to the Readme. I try to follow this style:

You can download the favicon 1024x1024 her Sir..https://drive.google.com/drive/folders/1RWT4gfUjfuaKINujYZcl1aMiVDmBKaOu
I'll make a PR now and upload the other files

created a PR #43 Sir, Thanks for accepting my proposal.. If you need any help regarding some modifications regarding logo in the future you can notify me. :) best regards!


Any plans for other popular model/methods ?

Mmdetection is a really excellent project for study & research , and kind enough for rookies .
We concern wether more fresh method/network/idel will be upadted here ?
If so , I believe this work will become Chinese Detectron !!!!! : )

Thanks. We have plans to add more methods, but not become the "Chinese Detectron".


Better Readme

Instructions to set API keys;
Instructions to build the project;
Instructions to run tests;

Closed by #d25a22c


add preview for publish to Azure development container

Adding "Preview" next to "Publish to new Azure SQL server local development container" option in the publish sql project dialog:

general SQL (160/150/etc) projects publish to container does not have the preview:

Pull Request Test Coverage Report for Build<PHONE_NUMBER>

1 of 1 (100.0%)  changed or added relevant line in 1 file are covered.
1 unchanged line in 1 file lost coverage.
Overall coverage increased (+0.008%) to 41.949%

Files with Coverage Reduction
New Missed Lines
%

extensions/sql-bindings/src/common/azureFunctionsUtils.ts
1
80.65%

Totals

Change from base Build<PHONE_NUMBER>:
0.008%

Covered Lines:
28929

Relevant Lines:
64064

💛  - Coveralls


facets: fix nested facets search results

INSPIR-3162

can you also please add a test to make sure this doesn't happen again in the future?


ERROR: environment already initialized to privatechatgpt - How to redeploy the solution?

I have successfully deployed the solution, however I want to redeploy the solution in a different subscription but when I use the command azd init or azd init -t microsoft/azurechat, I get the following error: ERROR: environment already initialized to privatechatgpt, how can I create a new environment to redeploy the solution again? Any help is appreciated, thanks.

Hi, you can run azd init with the "--environment" and/or "--subscription" options.  See the doco  for more details:

Thanks for the help David. Another question I had was, if I wanted to deploy the solution in a customer environment, how would I go about doing that? What access will I need? Any help is appreciated


Unknown result backend: 'redis://'

After solving #124 I found another error when running:
# python -m notifico init
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/tmp/notifico/notifico/__main__.py", line 49, in <module>
    sys.exit(main(sys.argv))
  File "/tmp/notifico/notifico/__main__.py", line 38, in main
    app = create_instance()
  File "notifico/__init__.py", line 104, in create_instance
    from notifico.views.account import account
  File "notifico/views/account/__init__.py", line 19, in <module>
    from notifico.services import reset, background
  File "notifico/services/background.py", line 7, in <module>
    @celery.task
  File "/usr/lib/python2.7/dist-packages/celery/app/__init__.py", line 191, in task
    return inner_create_task_cls(**options)(*args)
  File "/usr/lib/python2.7/dist-packages/celery/app/__init__.py", line 171, in _create_task_cls
    base = options.pop("base", None) or self.Task
  File "/usr/lib/python2.7/dist-packages/kombu/utils/__init__.py", line 262, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/usr/lib/python2.7/dist-packages/celery/app/__init__.py", line 197, in Task
    return self.create_task_cls()
  File "/usr/lib/python2.7/dist-packages/celery/app/__init__.py", line 95, in create_task_cls
    class Task(BaseTask):
  File "/usr/lib/python2.7/dist-packages/celery/app/__init__.py", line 98, in Task
    backend = self.backend
  File "/usr/lib/python2.7/dist-packages/kombu/utils/__init__.py", line 262, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/usr/lib/python2.7/dist-packages/celery/app/base.py", line 386, in backend
    return self._get_backend()
  File "/usr/lib/python2.7/dist-packages/celery/app/base.py", line 341, in _get_backend
    backend_cls = get_backend_cls(backend_cls, loader=self.loader)
  File "/usr/lib/python2.7/dist-packages/celery/utils/functional.py", line 48, in _M
    value = fun(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/celery/backends/__init__.py", line 31, in get_backend_cls
    "Did you spell it correctly?  (%s)" % (backend, exc))
ValueError: Unknown result backend: 'redis://'.  Did you spell it correctly?  (Couldn't import 'redis://': Empty module name)

The next patch solved the problem:
diff --git a/notifico/services/background.py b/notifico/services/background.py
index 85c72b2..0261fef 100644
--- a/notifico/services/background.py
+++ b/notifico/services/background.py
@@ -4,7 +4,7 @@ from flask.ext.mail import Message
 from notifico import create_instance, celery, mail

<EMAIL_ADDRESS>+celery.task
 def send_mail(*args, **kwargs):
     """
     Sends an email using Flask-Mail and Notifico's configuration

Now, notifico is working fine in our server - thanks a lot @TkTech for this.
Regards,
Carlos Ruiz
http://wiki.idempiere.org/en/User:CarlosRuiz

This is not a solution, @clery.task is a decorator, your patch is equal to removing the decorator from this function.
Try changing CELERY_RESULT_BACKEND in config.py to redis:
BROKER_URL = 'redis://'
CELERY_RESULT_BACKEND = 'redis'
CELERY_IMPORTS = ('notifico.services.background',)
CELERY_TASK_SERIALIZER = 'json'

I am not sure why you get this problem, since I am not familiar with celery. @TkTech  ?

Thanks @Dav1dde you're right, then the correct patch seems to be:
diff --git a/notifico/config.py b/notifico/config.py
index c353567..16412dc 100644
--- a/notifico/config.py
+++ b/notifico/config.py
@@ -65,7 +65,7 @@ SENTRY_DSN = None
 # Celery Configuration
 # ---
 BROKER_URL = 'redis://'
-CELERY_RESULT_BACKEND = 'redis://'
+CELERY_RESULT_BACKEND = 'redis'
 CELERY_IMPORTS = ('notifico.services.background',)
 CELERY_TASK_SERIALIZER = 'json'

I am not sure why it broke out of the sudden and if this is the correct fix, according to the documentation celery has two methods of creating the backend, get_backend_cls is used instead of get_backend_by_url (which worked before).
For now you do not need to patch notifico, simply override the value in your local_config.py.

This isn't really a bug - different versions and combinations of redis and celery accept different default strings for the CELERY_RESULT_BACKEND. We can always make the default more explicit which should always work.


A higher result

Hello ！I have a new problem that the resnet50 architecture get 96.1% ish accuracy twice! I set the parameters as what you mentioned in your paper. I keep all the default settings and only change the num_models to 0 and learning_rate in second stage to 1e-5. Did you get such high results in your runs?

30 epochs for each stage, and takes about 1 day in total.

I'm using the ModelNet40.


Add tooltips to graphviz

[ ] Closes #7970
[x] Tests passed
[x] Passes black dask / flake8 dask / isort dask

Can one of the admins verify this patch?

This is a great start!
I'd like to see you add some more information to the tooltip. You don't have to wait to be told what to do in https://github.com/dask/dask/issues/7970, you could be proactive and put together 2 or 3 proposed examples/demonstrations and show people what it would look like. Sometimes it can be easier to get feedback from a wide variety of people for something that already exists. (Of course, if you're busy working away on HTML representations, that's also a good use of your time)

Updates
I added attributes over from the collection_annotations into the tooltips and it looks nice. It actually is very similar to the HTML repr!
Arrays

DataFrames

I am not sure what other information would be useful at each layer's stage.

Suggestion: If possible, let's avoid the abbreviation "N" and instead write "Number of ..." if there's enough space. Abbreviations can be confusing.
It might be confusing for people to see "Dataframe type: pandas.core.DataFrame", we might like to try and be clearer that it's a dask dataframe made up of pandas partitions. (I don't know what's better here. "Dataframe partition type"? "Dask partition type"? Something else?)

I am not sure what other information would be useful at each layer's stage.

Perhaps @martindurant or others have suggestions. I don't think there's much other information available to add, and in any case we probably want to keep the tooltips small-ish.

Full sentences make it harder to skim read the information in the tooltip. Can we move back to the table-like tooltip, just without abbreviating words?
I'm excited about this, it's going to be a really handy feature to have.

Full sentences make it harder to skim read the information in the tooltip

Right! I agree.
New Changes
Mixed table-like and sentence-like display. Kept the sentence structure for the layer_type and n_tasks and used dictionary structure for the rest.

One idea: We could probably add some kind of warning or note for certain types of layer (or layer_types). This also works for
On top of my mind,

If the layer_type is DataFrameIO or Shuffle, we show some kind of note?
If the number of tasks is too large, we show it some way?
If it's a certain type of dataframe_type, we show tips about using it more efficiently?
3rd point but for array chunk type?

or we could just keep it as I currently have it. It looks minimal and clean.

Adding tooltips would close the gap between the HTML and the Graphical Representation of a High-Level Task Graph.
:rocket:

I like minimal and clean!

yes, of course, I am in favour of distinguishing the layer types, but at this point we add this information in the tooltip anyway. It may be nice to show it directly (in the other discussion)
we show the number of tasks in an obvious way, and give the absolute number too: no need for some arbitrary threshold fo issuing warnings
3., 4. this is not the place to give usage tips, and it's far from obvious what we might suggest

I like minimal and clean too!

We'll need to add a test. I'd look for where the other graphviz tests are and put your new one in the same file next to them.
Maybe we can create a tiny high level graph, then check that:

A tooltip attribute exists for the nodes, and
The length of the text string inside the tooltips is not zero (I don't think we need to check what the text says, just whether there is content in there)

I can't remember if you've already done a bunch of work with pytest or not, so ask any questions you need. I know you definitely had it set up in your development environment. To select a single specific test to run, it goes like this (replace the filepath and test name with the ones you want):
pytest dask/path/to/test_file.py::name_of_test

Also, once you have a test done I'd say this PR is basically finished. Feel free to mark it not a draft whenever you'd like.

added a test :)

@GenevieveBuckley

Added a test (it passes :heavy_check_mark: )
Columns will only show up if the length of the string is less than 40.

Will send the final demo in sometime.

Final Demo
Array
Same as above
DataFrame
with many columns

with less columns

I think this PR is now ready to be merged.
@GenevieveBuckley @martindurant

I've left two comments, about checking the tooltip string length and the mindeps CI test.
Once those are done, I think this is good to merge.

the gpuCI test is unrelated. I think someone has to approve this PR.

@martindurant This PR is ready to be merged :rocket:

:yellow_heart:


Input size limits

Line length limit
Literal size limit
IMAP Command total size limit

1 needs to Riser::Stream update.
2 and 3 are recommended to be larger than DRb's load_limit.

Riser is updated by #6


Return asset's attributes as keys when aggregate=False

Addresses https://github.com/NCAR/intake-esm/issues/163#issuecomment-543899740
import intake
col_file = "https://raw.githubusercontent.com/NCAR/intake-esm-datastore/master/catalogs/pangeo-cmip6.json"

col = intake.open_esm_datastore(col_file)
query = dict(experiment_id='historical', table_id='Oyr',
                 variable_id='o2', grid_label='gn', member_id='r1i1p1f1')
cat = col.search(**query)

# Disable aggregations
In [21]: dsets_pp = cat.to_dataset_dict(zarr_kwargs={"consolidated": True}, aggregate=False)
--> The keys in the returned dictionary of datasets are constructed as follows:
        'activity_id.institution_id.source_id.experiment_id.member_id.table_id.variable_id.grid_label'

--> There will be 2 group(s)

In [22]: dsets_pp.keys()
Out[22]: dict_keys(['CMIP.CCCma.CanESM5.historical.r1i1p1f1.Oyr.o2.gn', 'CMIP.IPSL.IPSL-CM6A-LR.historical.r1i1p1f1.Oyr.o2.gn'])

Cc @matt-long


Use the anne-bonny account on gerrithub

You need a real account to login to gerrithub and receive notifications.
I've added the ssh key so that anne-bonny can login.
Signed-off-by: Jamie Lennox<EMAIL_ADDRESS>

Build failed.

http://logs.bonnyci.org/gate_github/BonnyCI/hoist/289/1491289882.27/bonnyci-run-gate-multinode : FAILURE in 2m 44s

Build failed.

http://logs.bonnyci.org/gate_github/BonnyCI/hoist/289/1491316449.09/bonnyci-run-gate-multinode : FAILURE in 2m 48s

Merge Failed.
This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.


Request to update examples in Protractor API docs for async/await

Some of the examples provided in Protractor API docs should be updated to include how to use the API methods with async/await.
http://www.protractortest.org/#/api
It will be helpful if async/await examples are provided for below API methods. I use some of the API methods extensively in my code, refactoring the code to make it work with async/await.

Filter
http://www.protractortest.org/#/api?view=ElementArrayFinder.prototype.filter

Map
http://www.protractortest.org/#/api?view=ElementArrayFinder.prototype.map

Each
http://www.protractortest.org/#/api?view=ElementArrayFinder.prototype.each

Sure. I'll add this to the list.

I'll work on it.

Closed with https://github.com/angular/protractor/pull/5081


Merging develop to master

Thank you @george-cm for the contribution.

Thank you. I'm glad I could make a small contribution to the project.
On Thu, Dec 29, 2022, 06:56 Flavio Garcia @.***> wrote:

Thank you @george-cm https://github.com/george-cm for the contribution.
—
Reply to this email directly, view it on GitHub
https://github.com/candango/automatoes/pull/115#issuecomment-1367071906,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABKAWAN57BZF4RWKKDUL4GLWPUKXPANCNFSM6AAAAAATLZETFQ
.
You are receiving this because you were mentioned.Message ID:
@.***>


test dependencies

With this commit, Travis simply ensures that the installer dependencies do resolve properly. I will add "real" tests for the commands later on (you can merge this before and I can create a separate pull request later on).

@xabbuh could you please remind me the status of this PR? I see that you worked hard creating unit tests and it's sad to see that they haven't been merged. I don't know what's left to do in this PR.

@javiereguiluz Actually, I sadly forgot about this PR. I'll do a rebase and check if everything is still working. Thanks for the reminder.

This can be reviewed again.

I didn't have the time to dig deeper into setting things up properly. So I'm closing here. However, I will keep the branch on my fork. So anyone willing to finish this PR is welcome to just work on that branch. With #161 we also still have an issue reminding us that we really need tests for the installer.


Air75 Read Support Confirmed

Confirmed following functionality with Air75:

Reading bootloader, hash matches other NuPhy products
Reading full-flash

Added Air75 part (another Air60 reference)
Added minimum Rust version (let-else not stabilized before Rust 1.65 w/ PR https://github.com/rust-lang/rust/pull/93628)

I will stress that I did not attempt to write to the device. This is the only keyboard I have on hand at the moment :sweat_smile:

@donn Great to hear! Thanks for testing, the report and the PR! ❤
I've been writing to my Air60 quite a bit recently, things work fine and reliably, but only as long as there's a working method to jump back into the ISP bootloader. A sinolink programmer comes in handy whenever this breaks though.
P.S. Kudos for creating https://github.com/donn/nudelta!


Patch ClerkAuthentication, Fix JWT token verification

I also added the picture_image_url field into the User model in this PR


Fix link to kubernetes release

@marcbachmann Thanks, but I actually forgot to push the tag.  It is pointing the right place.  I've pushed the tag now.  Thanks!

Also good. You're welcome.


Does not work

Hi,
I've been using your plugin called lir for a while and its really good!
Wanted to try this out but couldn't manage to make it work. I installed it using packer and initialized it like:lua require('lir.git_satus').setup()
Confirmed that plugin is downloaded by using packer list. But I get an error upon opening nvim.
Vim version: NVIM v0.5.0-dev+1362-g3cd688ff7
E5108: Error executing lua [string ":lua"]:1: module 'lir.git_satus' not found:
        no field package.preload['lir.git_satus']
        no file './lir/git_satus.lua'
        no file '/opt/homebrew/Cellar/luajit-openresty/20210510/share/luajit-2.1.0-beta3/lir/git_satus.lua'
        no file '/usr/local/share/lua/5.1/lir/git_satus.lua'
        no file '/usr/local/share/lua/5.1/lir/git_satus/init.lua'
        no file '/opt/homebrew/Cellar/luajit-openresty/20210510/share/lua/5.1/lir/git_satus.lua'
        no file '/opt/homebrew/Cellar/luajit-openresty/20210510/share/lua/5.1/lir/git_satus/init.lua'
        no file './lir/git_satus.so'
        no file '/usr/local/lib/lua/5.1/lir/git_satus.so'
        no file '/opt/homebrew/Cellar/luajit-openresty/20210510/lib/lua/5.1/lir/git_satus.so'
        no file '/usr/local/lib/lua/5.1/loadall.so'
        no file './lir.so'
        no file '/usr/local/lib/lua/5.1/lir.so'
        no file '/opt/homebrew/Cellar/luajit-openresty/20210510/lib/lua/5.1/lir.so'
        no file '/usr/local/lib/lua/5.1/loadall.so'

Thanks for the report.
Sorry, I was typo'ing!
The correct one is require'lir.git_status'.
Please check it.

Thank you!


Add EGA identifier for manual import of datasets

Enhancement
Add EGA identifier for manual import of datasets
Why: a researcher tried to add a dataset from a repository we don't have yet, via manual input. This now results into broken links.
How to get there

Go to https://backoffice.bibliodev.ugent.be/dataset/add
Register a dataset manually
Edit the Dataset details

Expected behaviour

When a researcher adds a dataset they can add EGA as an identifier under the field "Persistent identifier type"

[ ] In the dropdown an extra option EGA appears

After saving the changes, whenever an identifier and persistent identifier type have been added, you can click on the identifier in several places.

[ ] The URL is https://ega-archive.org/datasets/ (and would result into a structure like https://ega-archive.org/datasets/EGAD00000000001, for the registered dataset it links to https://ega-archive.org/datasets/EGAD00001011679)
[ ] You can click the link on the detail page
[ ] You can click the link on the overview page, inside the dropdown on the right

[out of scope] The persistent identifier is shown on the Biblio front office https://github.ugent.be/Universiteitsbibliotheek/biblio/issues/145

See related issue https://github.ugent.be/Universiteitsbibliotheek/biblio/issues/145


Fix Table columns un-resizable bug

Changes

Fixes #897

Before/After
Before:
https://github.com/iTwin/iTwinUI/assets/45748283/b7b837b9-83f1-4622-ac87-8948f35df42a
After:
https://github.com/iTwin/iTwinUI/assets/45748283/863d41b6-6aab-4684-a14a-b5237e5e6b89

The table columns become un-resizable when they are given a width that is a string. (e.g. "200px", 50%, etc.). This is because when doing the resize, the below code is executed:
https://github.com/iTwin/iTwinUI/blob/607593ef67fc59408284e0fa68a6d5e9c3c8c7d9/packages/itwinui-react/src/core/Table/hooks/useResizeColumns.tsx#L500
… and when header.width = "200px" for example, Number("200px") = nan, and when the new values is deemed invalid by:
https://github.com/iTwin/iTwinUI/blob/607593ef67fc59408284e0fa68a6d5e9c3c8c7d9/packages/itwinui-react/src/core/Table/hooks/useResizeColumns.tsx#L298-L307
… we early return and do no resize.
This PR makes sure to also handle widths passed to the column as strings.
Testing

Tested with the below code in the vite playground:

App.tsx
import { Table } from '@itwin/itwinui-react';
import { useMemo } from 'react';

const App = () => {
  const columns = useMemo(
    () => [
      {
        Header: 'Header',
        columns: [
          {
            id: 'id1',
            accessor: 'id1',
            Header: 'id1',
            width: '20%',
          },
          {
            id: 'id2',
            accessor: 'id2',
            Header: 'id2',
            width: '20%',
          },
          {
            id: 'id3',
            accessor: 'id3',
            Header: 'id3',
            width: '60%',
          },
        ],
      },
    ],
    [],
  );

  const data = [
    {
      id1: 'id1',
      id2: 'id2',
      id3: 'id3',
    },
    {
      id1: 'id1',
      id2: 'id2',
      id3: 'id3',
    },
    {
      id1: 'id1',
      id2: 'id2',
      id3: 'id3',
    },
  ];

  return (
    <>
      <Table
        isResizable
        columns={columns}
        data={data}
        emptyTableContent='empty'
      />
    </>
  );
};

export default App;

All existing unit tests pass
Added a unit test to confirm that resizing columns works with string units too.

Docs

Added changeset.

It would be a good idea to maybe have a quick video demo of before and after. I was trying to reproduce, code sandbox link is not working.

Sounds good. Updated the PR description with the "Before/After" screen recordings and also the App.tsx code that I used for the demos.


[opt](session variable) max_msg_size_of_result_receiver

fix [Err] 1105 - TTransportException, msg: MaxMessageSize reached
MaxMessageSize is used to limit the max size of one filed of thrift struct during deserialization, default value of MaxMessageSize is about 104MB,  eg. set max_msg_size_of_result_receiver = 204000000 can increase MaxMessageSize to 204 MB

run buildall

run buildall


Add a pausing checkpoint phase to the beginning of autoscale grow and shrink protocols

In order to ensure that 2PC sinks can correctly flush precommitted
data before autoscale begins, we add a final checkpoint round
immediately after stopping the world in resilience mode. This requires
that we add a mechanism for "pausing checkpoints", i.e. checkpoints
that do not immediately lead to starting a new checkpoint timer.
When a checkpoint is complete, we now use a promise to determine what to do next.
Ordinarily we start the next checkpoint timer. But in the case of a "pausing
checkpoint" initiated from outside this actor, we use the provided promise
instead. It will then be the responsibility of whoever initiated the pausing
checkpoint to initiate the next one. In the case of autoscale, we wait to initiate the
next checkpoint until after the entire autoscale protocol is complete and we resume
message processing.

Can you describe the conditions under which the checkpoints don't resume?
Were there any autoscale events? How many workers had crashed?  Had the
sink crashed?  Were we past the 2nd checkpoint?
On Fri, Nov 1, 2019 at 9:43 PM Scott Lystig Fritchie <
<EMAIL_ADDRESS>wrote:

@slfritchie requested changes on this pull request.
Sorry about the delay in reviewing! Before task switching, I had
discovered that master branch back then, plus the 3 remaining
post-omnibus2 PRs, plus this PR resulted in checkpoints not resuming after
a crash event.
I've simplified a reproducing test case significantly:

Start with this PR's branch
Merge the effectively-once-script-update branch (used for PR #3038
https://github.com/WallarooLabs/wallaroo/pull/3038)
Use the master-crasher.sh recipe below.

... and checkpoints will not resume after the sink is disconnected and
reconnected.
$ cd /top/of/wallaroo/repo
$ git checkout autoscale-checkpoint
$ git checkout -b tmp-delete-me
$ git merge origin/effectively-once-script-update
$ dd if=testing/data/market_spread/nbbo/r3k-symbols_nbbo-fixish.msg bs=1000000 count=4 | od -x | sed 's/^/T/' > /tmp/input-file.txt
$ cd testing/correctness/scripts/effectively-once
$ make -C ../../../..     PONYCFLAGS="--verbose=1 -d -Dresilience -Dtrace -Dcheckpoint_trace -Didentify_routing_ids"     build-examples-pony-passthrough build-testing-tools-external_sender     build-utils-cluster_shrinker build-utils-data_receiver     build-testing-tools-fixed_length_message_blaster
$ ./master-crasher.sh 1 crash-sink ; echo "args of '2 crash-sink' will screw up checkpoints also"
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/WallarooLabs/wallaroo/pull/3044?email_source=notifications&email_token=ABEC3TGTQQYQ6QIS67AZFATQRSIGFA5CNFSM4JCEGEG2YY3PNVWWK3TUL52HS4DFWFIHK3DMKJSXC5LFON2FEZLWNFSXPKTDN5WW2ZLOORPWSZGOCKCLFRQ#pullrequestreview-310686406,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ABEC3THIXXVPGRLPH47K4RTQRSIGFANCNFSM4JCEGEGQ
.

Hi, sorry, I should've included a full set of logs with that comment. See http://wallaroolabs-dev.s3.amazonaws.com/logs/logs.1572901842.tar.gz

In the logs.1572901842.tar.gz, the initializer worker (in a 1-node system) has its connector sink crash roughly 6 seconds after the start of the test.  Taking a closer look:
1572901804.031218,INFO,ConnectorSink,ConnectorSink connection closed, throttling
1572901804.031224,INFO,ConnectorSink,ConnectorSink is experiencing back pressure, connected = false
1572901804.031230,INFO,ConnectorSink,RE-Connecting ConnectorSink to <IP_ADDRESS>:7200
...
1572901804.041620,DEBUG,ConnectorSink,Rcvd msg at ConnectorSink
1572901804.041633,DEBUG,ConnectorSink,Sink sentv: not connected
1572901804.131614,DEBUG,ConnectorSink,ConnectorSink connecting
1572901804.131656,INFO,ConnectorSink,Reconnection asio event
1572901804.131694,INFO,ConnectorSink,ConnectorSink connection failed
1572901804.131701,INFO,ConnectorSink,RE-Connecting ConnectorSink to <IP_ADDRESS>:7200
1572901805.005166,Initiating checkpoint 8 at 2019-11-04 21:10:05.5157
... skip until we get to the last log message:
1572901805.005591,_CheckpointEventLogPhase: check_completion() with 2 checkpointed and 6 total

At this point, Wallaroo hangs: there are no more attempts to reconnect to the sink, no more checkpoint activity, etc.
The connector-sink-fixes branch (PR #3042) contains more diagnostic info at the sink.  If I use the aloc_sink script from that branch onto this PR's branch, then I see that Wallaroo's sink reconnect attempt happens before the sink has been restarted; hence Wallaroo's complaint that a new sink connection fails is legit.  However, it's odd that there isn't a subsequent attempt by the sink to connect again later.
My guess right now is that the Pony runtime's backpressure is causing the deadlock.  How, exactly, isn't clear yet.  Note that PR #3042 disables use of runtime backpressure due to other deadlock complications.