[Firebase/Firestore][I-FST000001] Could not reach Cloud Firestore backend.

[REQUIRED] Step 1: Describe your environment

Xcode version: 11.4.1
Firebase SDK version: 6.24.0
Firebase Component: Firestore
Component version: 1.13.0
Installation method: CocoaPods

[REQUIRED] Step 2: Describe the problem
On both real devices and simulator, wifi and LTE, unable to load anything from the database with this issue:
2020-05-10 13:39:09.206329-0400 NFit30[2764:5183813] 6.24.0 - [Firebase/Firestore][I-FST000001] Could not reach Cloud Firestore backend. Backend didn't respond within 10 seconds.
 This typically indicates that your device does not have a healthy Internet connection at the moment. The client will operate in offline mode until it is able to successfully connect to the backend.

Read discussion surrounding similar issues here and here, with no success.
Relevant Code:
import Foundation
import Firebase
import SwiftUI
import Combine

class FirebaseSession: ObservableObject {

    //MARK: Properties
    @Published var workouts: [Workout] = []
    @Published var exercises: [Exercise] = []

    let db = Firestore.firestore()

    var didChange = PassthroughSubject<FirebaseSession, Never>()
    @Published var session: User? { didSet { self.didChange.send(self) }}
    var handle: AuthStateDidChangeListenerHandle?

    //MARK: Authentication functions

    func listen () {
        // monitor authentication changes using firebase
        handle = Auth.auth().addStateDidChangeListener { (auth, user) in
            if let user = user {
                // if we have a user, create a new user model
                print("Got user: \(user)")
                self.session = User(
                    uid: user.uid,
                    displayName: user.displayName,
                    email: user.email
                )
            } else {
                // if we don't have a user, set our session to nil
                self.session = nil
            }
        }
    }

    func signUp(
        email: String,
        password: String,
        handler: @escaping AuthDataResultCallback
    ) {

    }

    func signIn(
        email: String,
        password: String,
        handler: @escaping AuthDataResultCallback
        ) {
        Auth.auth().signIn(withEmail: email, password: password, completion: handler)
    }

    func signOut () -> Bool {
        do {
            try Auth.auth().signOut()
            self.session = nil
            return true
        } catch {
            return false
        }
    }

    func unbind () {
        if let handle = handle {
            Auth.auth().removeStateDidChangeListener(handle)
        }
    }

    //MARK: Workouts functions

    func getWorkouts() {
        print("test")
        db.collection("workouts").getDocuments() { (querySnapshot, err) in
            if let err = err {
                print("Error getting documents: \(err)")
            } else {
                self.workouts = []
                for document in querySnapshot!.documents {
                    print("\(document.documentID) => \(document.data())")
                    let workout = Workout(title: document.data()["title"] as! String, description: document.data()["description"] as! String, id: document.documentID, exercises: document.data()["exercises"] as! Array<String>, rounds: document.data()["rounds"] as! Int, breakLength: document.data()["breakLength"] as! Int)
                    self.workouts.append(workout)

                }
            }
        }
    }

    func getExercisesFromWorkout(workout: Workout) {
        exercises = []
        for id in workout.exercises {

            db.collection("exercises").document(id).getDocument { (querySnapshot, err) in
                if let err = err {
                    print("Error getting documents: \(err)")
                } else {
                    let exercise = Exercise(title: querySnapshot?.data()?["title"] as! String, tags: querySnapshot?.data()?["tags"] as! String, sku: querySnapshot?.data()?["sku"] as! String, id: id)
                    self.exercises.append(exercise)

                }
            }
        }
    }
}

class User {
    var uid: String
    var email: String?
    var displayName: String?

    init(uid: String, displayName: String?, email: String?) {
        self.uid = uid
        self.email = email
        self.displayName = displayName
    }

}

Hi Cameron,
a couple of notes:

You don't need to use PassthroughSubject any more to implement ObservableObject - check out https://peterfriese.dev/swiftui-firebase-fetch-data/ and https://peterfriese.dev/swiftui-firebase-codable/ for some pointers for how to simplify your code
Also, I think you can get rid of the User class. Instead, store the Firebase User directly. I've written about this in https://peterfriese.dev/replicating-reminder-swiftui-firebase-part2/

Regarding the error message you see - I've seen this before when inspecting a breakpoint and thus preventing Firestore from completing the call. Can you make sure this is not the case?

Thanks for the advice Peter, I'll check that stuff out. I can confirm it's not a breakpoint issue.

Hi cambardell,
Thanks for reporting the issue and sorry for the experience.
One common cause with this error is you are connecting behind a firewall or proxy. But this does not seem to be the case because it is failing with both LTE and WIFI.
Can I see the full debug logs? You can turn it one by Firestore.enableLogging(true).

Actually, it's working today. I didn't even change anything. I'll come back to this issue if it happens again. Thanks!


fix: Make sure the package can be built on Flutter web

What kind of change does this PR introduce?
Fixes an issue where the package is not able to build on Flutter web
Closes https://github.com/supabase/supabase-flutter/issues/989

Once we publish a new yet_another_json_isolate with this update, the currently retracted functions_client v2.3.0 will be restored.

Nice! Thanks for the fix @Vinzent03 !


Aviary viewer not viewing properly on Tanya's computer

Attaching screenshots.

Zoe and I are looking at the same project (her project) in Aviary. https://zillingworth.github.io/virtual-poem-on-screen/clip-of-claudia-rankine-for-wpr
She can see the right-hand panel but I cannot. My view is attached.

The problem appears to be a mismatch between how our iframe is sized and the way the Aviary player reacts to browser sizing.  I can reproduce this by changing this zoom level on my browser:


Goquery selection not finding element

I'm working on scraping a webpage to find contents of elements. I have a selector that successfully finds the element via JavaScript, but goquery fails to find the element.
I've made a gist that shows the issue, which can be found here: https://gist.github.com/migreva/bfda4fed23827fc9cbb99fd204de2da2
(specifically, the panic here)
Failing URL
https://www.whitehouse.gov/forreviewstaff-salaries
Failing Selector
body > *:nth-child(5) > *:nth-child(2) > *:nth-child(1) > *:nth-child(2) > *:nth-child(2) > *:nth-child(1) > *:nth-child(1)
Here's proof of successfully finding the element on that page via JavaScript

My apologies. Seems to be there's an element that gets added via JS, which is causing problems. Closing issue

Hello Michael,
This is because the document is altered by javascript - specifically, a <div id="nav-overlay"> is added inside the <section> element, which is why your selector works in javascript but not in goquery (because you only have the original page source, not the nodes added by the page via javascript).
The following selector works in goquery:
// note the :nth-child(1) instead of :nth-child(2) in the third part.
"body > *:nth-child(5) > *:nth-child(1) > *:nth-child(1) > *:nth-child(2) > *:nth-child(2) > *:nth-child(1) > *:nth-child(1)"

However, even though it works, I'd suggest a more robust selector unless this is a one-off thing. This will fail as soon as simple layout changes are made in the page. Something like #page h1 might be better.
Hope this helps,
Martin

Thanks Martin! That helps a bunch, I think I will make the selectors a little more specific to get around this issue


go: reader: make ordered iteration faster

Changelog

Added Message.PopulateFrom([]byte) error method to Message, which allows readers to avoid reallocating Message structs between messages.
Added MessageIterator.Next2(*Message) (*Schema, *Channel, *Message, error) method to the MessageIterator interface. This method allows the caller to re-use memory between message reads, and to avoid allocating a new Message on the heap for every message read.
optimized indexed reading to read much faster and use much less memory.

Docs

[x] Update docstrings
[x] Update examples

Description
This PR makes message iteration much faster. The included benchmark shows a pretty significant speedup:
Before
Note: the comparison benchmark based on main uses MessageIterator.Next() to read messages.
$ go test . -run=^$ -bench=BenchmarkReader -benchmem -benchtime=10x
goos: darwin
goarch: arm64
pkg: github.com/foxglove/mcap/go/mcap
BenchmarkReader/inorder/no_index-8                    10         532682221 ns/op         238.83 MB/s       7826155 msg/s        533306349 B/op   8054975 allocs/op
BenchmarkReader/inorder/index_file_order-8            10<PHONE_NUMBER> ns/op          67.29 MB/s       2204813 msg/s        909884028 B/op  12114339 allocs/op
BenchmarkReader/inorder/index_time_order-8            10<PHONE_NUMBER> ns/op          54.39 MB/s       1782145 msg/s        909889379 B/op  12114382 allocs/op
BenchmarkReader/inorder/index_rev_order-8             10<PHONE_NUMBER> ns/op          48.90 MB/s       1602346 msg/s        910261216 B/op  12114355 allocs/op
BenchmarkReader/inorder/bare_lexer-8                  10         196806788 ns/op         660.42 MB/s      21640757 msg/s        17005039 B/op       4672 allocs/op
BenchmarkReader/minor/no_index-8                      10         509497992 ns/op         241.92 MB/s       7927082 msg/s        531637254 B/op   8054932 allocs/op
BenchmarkReader/minor/index_file_order-8              10<PHONE_NUMBER> ns/op          66.97 MB/s       2194637 msg/s        909846889 B/op  12114373 allocs/op
BenchmarkReader/minor/index_time_order-8              10<PHONE_NUMBER> ns/op          54.82 MB/s       1796497 msg/s        909844632 B/op  12114332 allocs/op
BenchmarkReader/minor/index_rev_order-8               10<PHONE_NUMBER> ns/op          53.23 MB/s       1744292 msg/s        910212621 B/op  12114308 allocs/op
BenchmarkReader/minor/bare_lexer-8                    10         195830417 ns/op         638.20 MB/s      20912477 msg/s        15341232 B/op       4655 allocs/op
BenchmarkReader/major/no_index-8                      10         510946658 ns/op         241.74 MB/s       7921189 msg/s        532934768 B/op   8054945 allocs/op
BenchmarkReader/major/index_file_order-8              10<PHONE_NUMBER> ns/op          66.35 MB/s       2174050 msg/s        909833931 B/op  12114348 allocs/op
BenchmarkReader/major/index_time_order-8              10<PHONE_NUMBER> ns/op          54.53 MB/s       1786987 msg/s        909836941 B/op  12114379 allocs/op
BenchmarkReader/major/index_rev_order-8               10<PHONE_NUMBER> ns/op          51.12 MB/s       1675101 msg/s        910215724 B/op  12114370 allocs/op
BenchmarkReader/major/bare_lexer-8                    10         198086167 ns/op         632.44 MB/s      20724011 msg/s        16635893 B/op       4661 allocs/op
PASS
ok      github.com/foxglove/mcap/go/mcap        248.060s

After
% go test . -run=^$ -bench=BenchmarkReader -benchmem -benchtime=10x
goos: darwin
goarch: arm64
pkg: github.com/foxglove/mcap/go/mcap
BenchmarkReader/inorder/no_index-8                    10         209814421 ns/op         596.62 MB/s      19550071 msg/s        17491784 B/op       6310 allocs/op
BenchmarkReader/inorder/index_file_order-8            10         340508775 ns/op         360.08 MB/s      11799088 msg/s        10446040 B/op      16981 allocs/op
BenchmarkReader/inorder/index_time_order-8            10         341343088 ns/op         359.00 MB/s      11763672 msg/s        10443932 B/op      16955 allocs/op
BenchmarkReader/inorder/index_rev_order-8             10         348526088 ns/op         356.74 MB/s      11689775 msg/s         9996309 B/op      16964 allocs/op
BenchmarkReader/inorder/bare_lexer-8                  10         187405846 ns/op         664.57 MB/s      21776806 msg/s        17439823 B/op       4674 allocs/op
BenchmarkReader/minor/no_index-8                      10         211110267 ns/op         587.06 MB/s      19236916 msg/s        16522652 B/op       6284 allocs/op
BenchmarkReader/minor/index_file_order-8              10         356903517 ns/op         336.77 MB/s      11035283 msg/s        10419253 B/op      17006 allocs/op
BenchmarkReader/minor/index_time_order-8              10         552369746 ns/op         218.44 MB/s       7157955 msg/s        10444996 B/op      17744 allocs/op
BenchmarkReader/minor/index_rev_order-8               10         555191658 ns/op         220.27 MB/s       7217936 msg/s         9971279 B/op      17665 allocs/op
BenchmarkReader/minor/bare_lexer-8                    10         194812112 ns/op         554.57 MB/s      18172347 msg/s        16473271 B/op       4670 allocs/op
BenchmarkReader/major/no_index-8                      10         211406192 ns/op         579.88 MB/s      19001450 msg/s        17365727 B/op       6291 allocs/op
BenchmarkReader/major/index_file_order-8              10         354124750 ns/op         347.12 MB/s      11374355 msg/s        10418725 B/op      16979 allocs/op
BenchmarkReader/major/index_time_order-8              10         566783688 ns/op         215.38 MB/s       7057431 msg/s        16452847 B/op      17690 allocs/op
BenchmarkReader/major/index_rev_order-8               10         563155871 ns/op         218.15 MB/s       7148236 msg/s        15986112 B/op      17699 allocs/op
BenchmarkReader/major/bare_lexer-8                    10         195610721 ns/op         633.25 MB/s      20750327 msg/s        17316992 B/op       4672 allocs/op
PASS
ok      github.com/foxglove/mcap/go/mcap        68.716s

For the unindexed message iterator, all of the difference comes from:

being able to re-use a Message struct between calls to Next()
switching from storing channels and schemas in maps to slices.
using and re-using an internal buffer for lexing MCAP records.

For the indexed message iterator, we do all of the same things plus:

We maintain a pool of decompressed chunk buffers, which are re-used after all of the messages from a given chunk are read out.
We no longer read message indexes from the file, choosing instead to read the chunk content and build our own message index in memory. This allows us to read files written without message indexes in order, and also reduces I/O, which in some cases is probably faster (slow network connections with large message index overheads)
we no longer use a heap to maintain order, instead we maintain a sorted array of unread message indexes. Every time we encounter a new chunk:

if this chunk does not overlap with the last, clear the message index array and write the new chunk's messages in. If they are already in order, do not sort. This makes ordered iteration over an in-order MCAP as fast as unordered iteration.
if the new chunk's messages are not in order, sort the new chunk's messages.
if the new chunk overlaps with the last, append the new chunk's messages, and sort all unread messages.

New API justification
The issues with Next(p []byte) (*Schema, *Channel, *Message, error) that caused me to explore alternatives are:

The buffer p is used to store the message data, but if it isn't big enough, a new buffer is allocated for every message. A common idiom is to return the newly allocated buffer so that the caller can take ownership of it and re-use it, but the current API doesn't allow that.
A new Message struct is allocated on the heap for every iteration. Even if the message goes out of scope on the next loop, this still causes significant work for the garbage collector to do.

This new function signature re-uses the message passed in, if one is passed in. If nil is used, it creates a new Message.

cc. @wkalt

Do you have any thought on simplifying this to,
ReadNext(msg *message.Message) error

and making the reuse of the message's data buffer implicit?
Secondly, can you quantify the cost of the schema/channel lookups relative to the data allocations (and heap iterations IIRC) and share how much the relative contribution of each is hurting us? My recollection is that the main costs are in allocations and in the heap, rather than in the lookups. The lookups can also be converted to slice lookups pretty easily, either with a dynamically sized slice or (worst case) the full set of 2^16 which would only take 512kb. I would guess the cost of the lookups if that were done should go to approximately zero.
Third, could you post comparisons of the iteration performance of the reader vs the lexer? IIRC the lexer is much faster, like maybe 5x on adrian's mcnasty.mcap file. I think the goal of the reader improvements should roughly be to match the performance of the lexer, so it would be useful to have that as a comparison.
Fourth, have you investigated whether it is possible to bypass the heap completely in the case of a fully-ordered chunk (which can be known from inspection of the index)? My hunch previously was that some significant savings may come from here.
Great work overall, will take a closer look/test it out shortly.

@james-rms I'm not sure if it's useful at all but this is the code I was working on when I was digging into this previously: https://github.com/wkalt/mcap/blob/task/golang-reader-perf-improvements/go/mcap/writer_test.go#L703
The test linked there will generate a sample file. This benchmark is the one I guess I was then using for comparisons:
https://github.com/wkalt/mcap/blob/task/golang-reader-perf-improvements/go/mcap/reader_test.go#L749
I don't remember unfortunately where I left things off, except that I broke the code and then didn't get it working again. It looks to me like the ideas that patch is implementing are, a) switch to a slice-based lookup for the channels/schemas, and b) allow the user to pre-allocate structures (I doubt preallocating schema/channel actually matters at all, since those are comparatively infrequent).
My overall reaction to this PR is that I suspect we can probably maintain something closer to a traditional "iterator" interface (modulo whatever is needed for message preallocation -- i.e still returning our schema/channel pointers) while still getting all the performance improvements we want, so that in the 99% case where files are ordered, the lexer and the reader are doing identical work and are equally fast. I'm not wild about the interface changes required in this specific patch. I'm not really a fan of SchemaAndChannelForID - I think it just pushes the work our code did previously onto the user and probably results in the same amount of aggregate lookups being done in actual use (possibly with the user maintaining their own, duplicate mappings). If feasible I would definitely prefer if we could keep returning those pointers and instead make this efficient by changing the backing structures to slices.

I guess my concrete thought is, can we make this the Next signature:
Next(*Message) (*Schema, *Channel, *Message, error)

and get performance identical to the lexer, with that being the only required new interface. I suspect the answer is yes -- and if so, that seems like a better outcome.

Regarding the signature - i've pushed an update which makes the API ReadInto(*Message) (*Schema, *Channel, error). i can't see a point returning the message pointer if it's always going to be the same as the passed-in pointer.
Regarding always looking up the schema and channel - this makes a noticeable difference to the benchmark. I tested 2d2d767e4 against a patch that skips lookup and always returns nil for schemas and channels:
Without lookup
go test ./... -run=^$ -bench=BenchmarkReader -benchmem -benchtime=20s
BenchmarkReader/no_index-8<PHONE_NUMBER>24 ns/op         149.31 MB/s       4892764 msg/s        18126126 B/op       2883 allocs/op
BenchmarkReader/with_index-8                  62         381960212 ns/op          74.79 MB/s       2450602 msg/s         7425817 B/op       5960 allocs/op
PASS

With lookup
go test ./... -run=^$ -bench=BenchmarkReader -benchmem -benchtime=20s
BenchmarkReader/no_index-8<PHONE_NUMBER>86 ns/op         132.22 MB/s       4332659 msg/s        16929751 B/op       2871 allocs/op
BenchmarkReader/with_index-8                  55         429351720 ns/op          71.90 MB/s       2356059 msg/s         7425896 B/op       5960 allocs/op
PASS

My idea when separating the lookup to its own function is that for users who need to go fast, they rarely want the schema and channel information more than once.

If you're splitting an MCAP up,  for the first message you figure out which channel IDs need to go into which stream, then you build up a mapping and key off the channel ID after then.
If you're trying to decode messages, you instantiate your deserializers with schema information once per channel, you store them in a mapping of channel ID -> deserializer, and you never look at the schema information again.

Also re: testing against the lexer: I have a rough estimate here:
BenchmarkReader/lexer-8                    10         175206304 ns/op         173.64 MB/s       5689735 msg/s        17305244 B/op       1260 allocs/op

@james-rms I think there is definitely value in returning the message pointer because some users may prefer to allocate on each call, i.e by passing in a nil as the input pointer.
This would be useful if they want to pass the messages through some kind of processing pipeline and aren't too concerned about maximal perf. I'm assuming that in a future version we'll want to deprecate & remove the old interface and push users toward this one.

Is that benchmark you posted for the lexer comparable to the other two posted earlier? That seems like a great basis for comparison

Here's my results fwiw:
[~/repos/mcap/go/mcap] (jrms/faster-message-iterator) $ go test ./... -run=^$ -bench=BenchmarkReader -benchmem -benchtime=20s
goos: linux
goarch: amd64
pkg: github.com/foxglove/mcap/go/mcap
cpu: AMD Ryzen 9 7950X 16-Core Processor
BenchmarkReader/no_index-32<PHONE_NUMBER>97 ns/op         163.32 MB/s       5351822 msg/s        17561395 B/op       2879 allocs/op
BenchmarkReader/with_index-32                 69         340445082 ns/op          89.79 MB/s       2942353 msg/s         7415722 B/op       5960 allocs/op
BenchmarkReader/lexer-32<PHONE_NUMBER>22 ns/op         338.66 MB/s       6576108 msg/s        17148071 B/op       1238 allocs/op
PASS
ok      github.com/foxglove/mcap/go/mcap        111.347s

Here's the code I added to the benchmark for the lexer. I believe it's comparable to what you must have done, though I'm getting 2x your throughput for the lexer case only so it's possible I'm doing something different. Curious what that would be:
	// lexer
	b.Run("lexer", func(b *testing.B) {
		b.StopTimer()
		buf := &bytes.Buffer{}
		writer, err := NewWriter(buf, &WriterOptions{
			Chunked:     true,
			Compression: CompressionZSTD,
		})
		require.NoError(b, err)
		channelCount := 200
		messageCount := uint64(1000000)
		messageSize := 32
		require.NoError(b, writer.WriteHeader(&Header{}))
		require.NoError(b, writer.WriteSchema(&Schema{ID: 1, Name: "empty", Encoding: "none"}))
		for i := 0; i < channelCount; i++ {
			require.NoError(b, writer.WriteChannel(&Channel{
				ID:              uint16(i),
				SchemaID:        1,
				Topic:           "/chat",
				MessageEncoding: "none",
			}))
		}
		contentBuf := make([]byte, messageSize)
		for i := uint64(0); i < messageCount; i++ {
			channelID := uint16(i % uint64(channelCount))
			_, err := rand.Read(contentBuf)
			require.NoError(b, err)
			require.NoError(b, writer.WriteMessage(&Message{
				ChannelID:   channelID,
				Sequence:    uint32(i),
				LogTime:     i,
				PublishTime: i,
				Data:        contentBuf,
			}))
		}
		require.NoError(b, writer.Close())
		b.StartTimer()
		for i := 0; i < b.N; i++ {
			s := time.Now()
			lexer, err := NewLexer(bytes.NewReader(buf.Bytes()))
			require.NoError(b, err)

			buf := make([]byte, 1024)
			var msgBytes uint64
			var readMessages uint64
			for {
				tokenType, token, err := lexer.Next(buf)
				if errors.Is(err, io.EOF) {
					break
				}
				require.NoError(b, err)

				if len(token) > len(buf) {
					buf = token
				}
				if tokenType == TokenMessage {
					readMessages++
					msgBytes += uint64(len(token))
				}
			}
			b.ReportMetric(float64(messageCount)/time.Since(s).Seconds(), "msg/s")
			b.ReportMetric(float64(msgBytes)/(time.Since(s).Seconds()*1024*1024), "MB/s")
			require.Equal(b, messageCount, readMessages)
		}
	})

this visualization of CPU consumption in the indexed case puts >33% of the time in heap operations. For the specific file we are looking at, messages are ordered so heap is technically not needed.
We also have ~8.6% of time in map operations. I suspect that's mostly from the channel/schema lookups in NextInto:
(pprof) list NextInto
Total: 26.27s
ROUTINE ======================== github.com/foxglove/mcap/go/mcap.(*indexedMessageIterator).NextInto in /home/wyatt/repos/mcap/go/mcap/indexed_message_iterator.go
     890ms     15.30s (flat, cum) 58.24% of Total
      20ms       20ms    284:func (it *indexedMessageIterator) NextInto(msg *Message) (*Schema, *Channel, error) {
      40ms       40ms    285:   if !it.hasReadSummarySection {
         .       60ms    286:           err := it.parseSummarySection()
         .          .    287:           if err != nil {
         .          .    288:                   return nil, nil, err
         .          .    289:           }
         .          .    290:           // take care of the metadata here
         .          .    291:           if it.metadataCallback != nil {
         .          .    292:                   for _, idx := range it.metadataIndexes {
         .          .    293:                           _, err = it.rs.Seek(int64(idx.Offset), io.SeekStart)
         .          .    294:                           if err != nil {
         .          .    295:                                   return nil, nil, fmt.Errorf("failed to seek to metadata: %w", err)
         .          .    296:                           }
         .          .    297:                           opcode, data, err := readRecord(it.rs)
         .          .    298:                           if err != nil {
         .          .    299:                                   return nil, nil, fmt.Errorf("failed to read metadata record: %w", err)
         .          .    300:                           }
         .          .    301:                           if opcode != OpMetadata {
         .          .    302:                                   return nil, nil, fmt.Errorf("expected metadata record, found %v", data)
         .          .    303:                           }
         .          .    304:                           metadata, err := ParseMetadata(data)
         .          .    305:                           if err != nil {
         .          .    306:                                   return nil, nil, fmt.Errorf("failed to parse metadata record: %w", err)
         .          .    307:                           }
         .          .    308:                           err = it.metadataCallback(metadata)
         .          .    309:                           if err != nil {
         .          .    310:                                   return nil, nil, fmt.Errorf("metadata callback failed: %w", err)
         .          .    311:                           }
         .          .    312:                   }
         .          .    313:           }
         .          .    314:   }
         .          .    315:
      60ms       60ms    316:   for it.indexHeap.len() > 0 {
      10ms      8.70s    317:           ri, err := it.indexHeap.Pop()
      10ms       10ms    318:           if err != nil {
         .          .    319:                   return nil, nil, err
         .          .    320:           }
         .          .    321:           if ri.ChunkSlotIndex == -1 {
         .      3.91s    322:                   err := it.loadChunk(ri.chunkIndex)
         .          .    323:                   if err != nil {
         .          .    324:                           return nil, nil, err
         .          .    325:                   }
         .          .    326:                   continue
         .          .    327:           }
         .          .    328:           chunkOffset := ri.MessageOffsetInChunk
      40ms       40ms    329:           decompressedChunk := it.decompressedChunks[ri.ChunkSlotIndex]
      60ms       60ms    330:           length := binary.LittleEndian.Uint64(decompressedChunk.buf[chunkOffset+1:])
     120ms      120ms    331:           messageData := decompressedChunk.buf[chunkOffset+1+8 : chunkOffset+1+8+length]
      10ms       10ms    332:           existingbuf := msg.Data
      30ms      220ms    333:           if err := msg.PopulateFrom(messageData); err != nil {
         .          .    334:                   return nil, nil, err
         .          .    335:           }
     200ms      310ms    336:           msg.Data = append(existingbuf[:0], msg.Data...)
      30ms       30ms    337:           it.decompressedChunks[ri.ChunkSlotIndex].unreadMessages--
     150ms      1.11s    338:           channel, ok := it.channels[msg.ChannelID]
         .          .    339:           if !ok {
         .          .    340:                   return nil, nil, fmt.Errorf("message with unrecognized channel ID %d", msg.ChannelID)
         .          .    341:           }
     100ms      590ms    342:           schema, ok := it.schemas[channel.SchemaID]
         .          .    343:           if !ok && channel.SchemaID != 0 {
         .          .    344:                   return nil, nil, fmt.Errorf("channel %d with unrecognized schema ID %d", msg.ChannelID, channel.SchemaID)
         .          .    345:           }
      10ms       10ms    346:           return schema, channel, nil
         .          .    347:   }
         .          .    348:   return nil, nil, io.EOF
         .          .    349:}

if we can switch to slices for that I bet it would get cheaper.
On the memory side:

we're doing a fair amount of allocation in heapPush, which is just appending to a slice. I'm not sure how to make it convenient for the code, but we should have everything we need to preallocate the indices array.
Another thought is that if this can be converted to a merge of sorted lists, then only the top element of each list needs to be in the heap at a time, if a heap is used. That would mean we don't need to aggregate big indices lists (responsible for 18% of allocations here). I think this would be applicable in the case of no overlapping channels.
Annoyingly this picture also includes allocations from the test setup, accounting for > 40%. If I figured out how to drop that stuff out then the contribution of the indices appends would grow.

Here's the updated results after decrementing 22 bytes from my message bytes increment, noted by James.
[~/repos/mcap/go/mcap] (jrms/faster-message-iterator) $ go test ./... -run=^$ -bench=BenchmarkReader -benchmem -benchtime=20s -memprofile mem.prof -cpuprofile cpu.prof
goos: linux
goarch: amd64
pkg: github.com/foxglove/mcap/go/mcap
cpu: AMD Ryzen 9 7950X 16-Core Processor
BenchmarkReader/no_index-32<PHONE_NUMBER>58 ns/op         165.97 MB/s       5438499 msg/s        17651637 B/op       2879 allocs/op
BenchmarkReader/with_index-32                 68         343562631 ns/op          88.49 MB/s       2899692 msg/s         7428013 B/op       5956 allocs/op
BenchmarkReader/lexer-32<PHONE_NUMBER>51 ns/op         197.77 MB/s       6480498 msg/s        17214019 B/op       1239 allocs/op
PASS
ok      github.com/foxglove/mcap/go/mcap        111.862s

my results on latest code:
[~/repos/mcap/go/mcap] (jrms/faster-message-iterator) $ go test ./... -run=^$ -bench=BenchmarkReader -benchmem -benchtime=20s -cpuprofile cpu.prof -memprofile mem.prof
goos: linux
goarch: amd64
pkg: github.com/foxglove/mcap/go/mcap
cpu: AMD Ryzen 9 7950X 16-Core Processor
BenchmarkReader/no_index-32         	     151	 157370167 ns/op	 192.06 MB/s	   6293340 msg/s	17173000 B/op	    2866 allocs/op
BenchmarkReader/index_file_order-32 	     148	 161401921 ns/op	 195.04 MB/s	   6391049 msg/s	 6091088 B/op	    5971 allocs/op
BenchmarkReader/index_time_order-32 	     148	 161096652 ns/op	 192.76 MB/s	   6316549 msg/s	 6090255 B/op	    5960 allocs/op
BenchmarkReader/index_rev_order-32  	     150	 159409812 ns/op	 191.95 MB/s	   6289894 msg/s	 5306228 B/op	    5965 allocs/op
BenchmarkReader/bare_lexer-32       	     154	 151403426 ns/op	 213.52 MB/s	   6996691 msg/s	17123125 B/op	    1243 allocs/op
PASS
ok  	github.com/foxglove/mcap/go/mcap	199.761s

Would it be sensible at all to call this Next2 instead of NextInto? I understand "Into" conveys something but maybe that is also conveyed by the arg type.
The benefit to me seems like Next2 -> Next is a more natural change when major V2 is published, than NextInto -> Next.

a possible drawback is once we call something Next2, we better release V2 before we need a Next3. But I think that's not really different from NextInto. We'd look equally silly if we had Next, NextInto, NextIntoForReal, etc.

Re-posting some thoughts I shared with James in Slack:
I don't have many strong feelings on golang things but I think ya'll can do better than Next2 as an API name to live alongside the existing api. If it isn't possible to re-use the existing name with overloading, then something like NextInto(...) as James proposed sounded reasonable to me. I cannot recall exactly but I have used c/c++ like apis where you could either pass an existing object or null if you wanted one allocated for you. Seems like a sane pattern.

The object or null approach is the one we've taken. I obviously disagree on the naming but won't block you guys. I think NextInto is kind of a dead end. What do you call the thing after that? What do you do when we ship a V2 and there is no more Next, only NextInto?
Just seems kinda strange but again, not a dealbreaker to me.

another option to consider, would be to simply roll a major. I wonder if there's a good reason to avoid that?

Final minor suggestion - I think we should change "major" and "minor" in these benchmark names to "major_overlap" and "minor_overlap" for clarity.
Anyway, final results from me:
BenchmarkReader/inorder/no_index-32<PHONE_NUMBER>17 ns/op         609.76 MB/s      19980710 msg/s        18784082 B/op       6300 allocs/op
BenchmarkReader/inorder/index_file_order-32<PHONE_NUMBER>85 ns/op         677.58 MB/s      22202914 msg/s        10451106 B/op      16981 allocs/op
BenchmarkReader/inorder/index_time_order-32<PHONE_NUMBER>38 ns/op         675.79 MB/s      22144266 msg/s        10451253 B/op      16989 allocs/op
BenchmarkReader/inorder/index_rev_order-32<PHONE_NUMBER>67 ns/op         647.86 MB/s      21229217 msg/s         9993184 B/op      16985 allocs/op
BenchmarkReader/inorder/bare_lexer-32<PHONE_NUMBER>39 ns/op         600.44 MB/s      19675251 msg/s        18732559 B/op       4671 allocs/op
BenchmarkReader/minor/no_index-32<PHONE_NUMBER>48 ns/op         499.60 MB/s      16371129 msg/s        15390843 B/op       6275 allocs/op
BenchmarkReader/minor/index_file_order-32<PHONE_NUMBER>32 ns/op         673.93 MB/s      22083281 msg/s        10418086 B/op      16978 allocs/op
BenchmarkReader/minor/index_time_order-32             68         362007549 ns/op         347.44 MB/s      11384992 msg/s        10447251 B/op      17721 allocs/op
BenchmarkReader/minor/index_rev_order-32              69         355197195 ns/op         353.53 MB/s      11584413 msg/s         9977055 B/op      17704 allocs/op
BenchmarkReader/minor/bare_lexer-32<PHONE_NUMBER>11 ns/op         586.82 MB/s      19229078 msg/s        15341271 B/op       4651 allocs/op
BenchmarkReader/major/no_index-32<PHONE_NUMBER>59 ns/op         550.78 MB/s      18048103 msg/s        16619558 B/op       6283 allocs/op
BenchmarkReader/major/index_file_order-32<PHONE_NUMBER>90 ns/op         671.87 MB/s      22016043 msg/s        10418646 B/op      16992 allocs/op
BenchmarkReader/major/index_time_order-32             63         371470199 ns/op         342.28 MB/s      11215845 msg/s        16457851 B/op      17714 allocs/op
BenchmarkReader/major/index_rev_order-32              66         356158503 ns/op         356.68 MB/s      11687706 msg/s        15990102 B/op      17716 allocs/op
BenchmarkReader/major/bare_lexer-32<PHONE_NUMBER>03 ns/op         633.73 MB/s      20766082 msg/s        16569771 B/op       4656 allocs/op
PASS
ok      github.com/foxglove/mcap/go/mcap        513.643s

Behavior is close on files with small overlap, and we still have a gap with heavy overlap as expected. Looks great!

I wonder if there's a good reason to avoid that

In golang all dependencies of a project have to use the same version? So if you make API incompatible changes then some dependency that also needs the mcap library will break if not updated to the latest? This is why golang uses the folder major versioning approach iirc? I have not done enough golang to say if its better to roll the major or adopt the 2 strategy. Can you point to examples of popular libraries that have added 2 to their APIs rather than rolling a major? Or some other approach?

@defunctzombie I'm not sure I'm quite following - hopefully this will be helpful -

All dependencies of a project in go do not need to use the same version. They can use whatever tagged version they desire.
Dependency updates are opt-in, so shipping a new version does not force any upgrade on users.
There is no distinction between releasing a major version of the lib and adopting the /v2 suffix on import paths. The way to release a major would be to bump the major version in the tag, and do the /v2 suffix. So it's not a "rather than" - you'd do both.
There are instructions here: https://go.dev/wiki/Modules#releasing-modules-v2-or-higher . Of the two approaches mentioned, you want to use the first one. The second approach was only relevant during a transitory period during adoption of modules, for projects that needed to support older versions of go.
An example of a go project on a version greater than 1 is https://github.com/pierrec/lz4
Because MCAP has released a 1.y.z already, and because you guys like semver, you don't have a choice about the v2 suffixing. If you bump the tag without suffixing the import path, the new tag won't be consumable.

While it isn't an option for MCAP, numerous projects choose to ignore SIV (semantic import versioning, this suffixing stuff) for various reasons. The way to do this is to adopt a convention that isn't real semver, such as keeping a permanent 0 or 1 major version and communicating breaking changes in other ways. Foxglove has dependencies that take this approach as well. I don't recall which offhand (protobuf maybe?) but I expect you might find a few if you executed go get -u in the data platform project and run the tests. I generally have no expectation that my dependencies adhere to semver, whether or not they claim to, so I consider this fine as well.
The point of SIV is to allow consumers to use multiple major versions of a lib at the same time. That's not a requirement I've ever encountered so to me personally, it's an inconvenience to update paths, docs, etc just for a version bump.

I think a good argument against rolling a major update into this PR, is that in order to avoid update churn it would be best if the major were specifically planned so that any breaking changes we want to put in can be bundled together. Due in part to SIV, multiple major upgrades in a row is annoying for consumers.


docs: Translation of Runes into Portuguese

@raphjaph
You could do a little review. I am a rune protocol and ordinals enthusiast and would like to share the knowledge in my native language. Thank you for your attention.

Thanks for the translation!
Unfortunately I can't merge this because it does not conform to our translation template. Please have a look at it here: https://github.com/ordinals/ord#translations. Then you can open a new PR with the correct template.

Thanks for the tip. I will redo it following the specified rules


ADMIN_DOWN should also not cause parent_pending=1

Currently, only OFFLINE is the status that removes that parent cache from being considered in the child cache's parent_pending key in that child cache's /update/. This is important as ORT checks the parent_pending key before processing updates for that cache.
This is the code that likely needs to change in lib/UI/Server.pm:
			while ( my $prow = $rs_parents->next ) {
				if (   $prow->upd_pending == 1
					&& $prow->status->name ne "OFFLINE" )
				{
					$parent_pending{ $rs_servers->single->host_name } = 1;
				}
			}

not sure if i agree... ADMIN_DOWN hosts run ort? This only works if the transition from ADMIN_DOWN to REPORTED or ONLINE forces an ort badass?

Hmm, yes, they should. Let me think about that -- perhaps we should more aggressively put caches into an OFFLINE status (with some automation?)? A quick reboot can cause an ORT run to be missed, and propagation delays because of it.

Closing issue -- @knutsel is right -- the operations procedure should be such that if a cache cannot successfully complete a 'syncds' ORT run, it should be in OFFLINE.


About Generate Image from GT Representation resault

Dear author, in the GT-generated images, the images containing humans do not seem to produce very good results. Is there any problem with my implementation? Looking forward to your reply, thanks.

Yes, human is a common problem when generating ImageNet images, especially human faces. If only generating human is with poor performance and other generation looks reasonable, your implementation should be correct.


Refactor webhook by CLI

this refactors the cli code for webhook by generalising a bit
to allow other provider add easily
Changes

Submitter Checklist

[ ] ♽  Run make test lint before submitting a PR (ie: with pre-commit, no need to waste CPU cycle on CI
[ ] 📖 If you are adding a user facing feature or make a change of the behavior, please verify that you have documented it
[ ] 🧪 100% coverage is not a target but most of the time we would rather have a unit test if you make a code change.
[ ] 🎁 If that's something that is possible to do please ensure to check if we can add a e2e test.
[ ] 🔎 If there is a flakiness in the CI tests then don't necessary ignore it, better get the flakyness fixed before merging or if that's not possible there is a good reason to bypass it. (token rate limitation may be a good reason to skip).

@chmouel refactored the code a bit to add other providers
ptal :)

there is an issue with the e2e test we need to figure before merging prs...
not quite sure what is yet but we have crash in the logs

/retest

let's if snazy breaks :-p


Does this work two way?

I'd like to make some trello cards appear in VSTS, does this allow me to do that?

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: 1402b59d-915f-2457-e198-60898fd66191
Version Independent ID: af8951ef-3bc1-8748-81a5-03696c6f5e85
Content: Trello with Azure DevOps Services
Content Source: docs/service-hooks/services/trello.md
Product: devops
GitHub Login: @elbatk
Microsoft Alias: elbatk

Hey @JawadS - Service Hooks is one direction. It sends updates out to Trello and doesn't take them in. I'm not aware of how to get Trello cards to appear in Azure DevOps (formerly VSTS) at this time.

Hey @JawadS - Service Hooks is one direction. It sends updates out to Trello and doesn't take them in. I'm not aware of how to get Trello cards to appear in Azure DevOps (formerly VSTS) at this time.

I've had to do this using a Zap (Zapier) eventually. But thanks for the help!


If i am using carbon, the  RustServerMetrics.dll harmony mod keeps my server in infinite loading and does not load

If i am using carbon, the  RustServerMetrics.dll harmony mod keeps my server in infinite loading and does not load but on removing the mod from the harmony folder in carbon directory server loads normally, when not loading it creates the HarmonyMods_Data and the config file normally but the server does not load.
i didnt had the issue when on oxide.
any help?

so should i contact carbon people to make it work like them adding support ?

You can try. As far as I know, the owner of this repo doesn't mind if someone creates pull requests.
https://github.com/Pinkstink-Rust/Rust-Server-Metrics/issues/11

will ask them if they can do anything because this helps me dealing with ddos attacks on my server and also checking what is causing server lag

As Monah said, Carbon isn't currently supported and no current plans to add support from the maintainers. PRs always welcome, though.
Closing for now.


unstructured induction tactic is unusable?

Prerequisites

[X] Put an X between the brackets on this line if you have done all of the following:

Checked that your issue isn't already filed.
Reduced the issue to a self-contained, reproducible test case.

Description
In this example:
theorem zeroAdd : 0 + x = x := by
  induction x
  rfl
  rw Nat.addSucc
  admit

Expected behavior:
Lean 3 generated accessible names there, but I suppose this is considered bad practice in Lean 4. But there also was a syntax to assign user names. It seems that the only way to solve this right now is a structured induction. I suppose this is because of a missing feature?
Actual behavior:
induction hypothesis and variable are inaccessible and there is no syntax to assign their names (aside from the structured version, that is).
Versions
Lean (version 4.0.0, commit 3bc5b89ac365, Release)
Additional Information
I've also seen this in injection.
BTW, in Emacs client this inaccessible notation now erases the cross and looks just like ordinary variable using the same font (lsp-mode bug?). This looks confusing.

In the unstructured version, one is supposed to finish the proof without referring to inaccessible variable names. Example:
theorem zeroAdd : 0 + x = x := by
  induction x
  rfl
  rw Nat.addSucc
  apply congrArg
  assumption

The example above is too convoluted, but it becomes more realistic when one is using more complex tactics.
One may still use the Lean3 approach
set_option hygienicIntro false in
theorem zeroAdd' : 0 + x = x := by
  induction x
  rfl
  rw Nat.addSucc
  rw v_0

One may also use the following approach where the "structure version" is used only for naming variables.
theorem zeroAdd'' : 0 + x = x := by
  induction x with | succ _ ih => _ | _ => _
  rfl
  rw Nat.addSucc
  rw ih

One can even add a nicer macro on top of it to hide the => _s

Ok, thanks for explaining!
I suppose that this issue is just about missing docs then. I'm closing this.


Smart open does not work with S3 buckets containing dots

Smart open is not working with buckets that contains . in name such as com.ziky90.project.
Particularly I am getting an error log that looks as follows.
Traceback (most recent call last):
  File "/home/ubuntu/bucket_accessor/model_loader.py", line 70, in <module>
    model = TModel.load(model_path)
  File "/usr/local/lib/python2.7/dist-packages/gensim/utils.py", line 252, in load
    obj = unpickle(fname)
  File "/usr/local/lib/python2.7/dist-packages/gensim/utils.py", line 906, in unpickle
    with smart_open(fname) as f:
  File "/usr/local/lib/python2.7/dist-packages/smart_open/smart_open_lib.py", line 93, in smart_open
    return S3OpenRead(parsed_uri)
  File "/usr/local/lib/python2.7/dist-packages/smart_open/smart_open_lib.py", line 191, in __init__
    self.read_key = s3_connection.get_bucket(parsed_uri.bucket_id).lookup(parsed_uri.key_id)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 502, in get_bucket
    return self.head_bucket(bucket_name, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 521, in head_bucket
    response = self.make_request('HEAD', bucket_name, headers=headers)
  File "/usr/local/lib/python2.7/dist-packages/boto/s3/connection.py", line 664, in make_request
    retry_handler=retry_handler
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 1071, in make_request
    retry_handler=retry_handler)
  File "/usr/local/lib/python2.7/dist-packages/boto/connection.py", line 943, in _mexe
    request.body, request.headers)
  File "/usr/lib/python2.7/httplib.py", line 1001, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python2.7/httplib.py", line 1035, in _send_request
    self.endheaders(body)
  File "/usr/lib/python2.7/httplib.py", line 997, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python2.7/httplib.py", line 850, in _send_output
    self.send(msg)
  File "/usr/lib/python2.7/httplib.py", line 812, in send
    self.connect()
  File "/usr/lib/python2.7/httplib.py", line 1212, in connect
    server_hostname=server_hostname)
  File "/usr/lib/python2.7/ssl.py", line 350, in wrap_socket
    _context=self)
  File "/usr/lib/python2.7/ssl.py", line 566, in __init__
    self.do_handshake()
  File "/usr/lib/python2.7/ssl.py", line 796, in do_handshake
    match_hostname(self.getpeercert(), self.server_hostname)
  File "/usr/lib/python2.7/ssl.py", line 269, in match_hostname
    % (hostname, ', '.join(map(repr, dnsnames))))
CertificateError: hostname 'com.ziky90.project.s3.amazonaws.com' doesn't match either of '*.s3.amazonaws.com', 's3.amazonaws.com'

NOTE: (I use smart_open from gensim)
It seems to me that the bug is related to https://github.com/boto/boto/issues/2836 and I am wondering if for example replacing boto by boto3  would help?

Thanks for reporting. Last time I looked, boto3 was unfinished and had many issues.
But if it works better, we could (should) switch to boto3 of course. This will need careful testing (and updating all code) though, as the two libraries are not API-compatible IIRC.

Similar problem for me. Perhaps smart_open isn't being used recursively at the contained objects within models like LsiModel (LsiModel.Projection object is split into two pickles, and smart_open not being used to open them separately). I have no problem opening these pickles with smart_open using the exact URIs given in the warning message.
```python
lsi = LsiModel.load(s3_uri('lsi_model_both_130K'))
WARNING:root:failed to load projection from<EMAIL_ADDRESS>[Errno 2] No such file or directory<EMAIL_ADDRESS>```

That's right, the numpy object are loaded using numpy's native load operation, which doesn't support s3 or smart_open.
The easiest work-around is to store your object using pickle (a single file) and smart_open, avoiding the load/save that split the object into multiple files. Or pass separately=[] to lsi.save(), that should work too.

Perfect! Hadn't thought of the single-file workaround. Thanks!

A solution to this issue would be to set calling_format=ProtocolIndependentOrdinaryCallingFormat() when calling smart_open and passing calling_format to boto.connect_s3. Additionally, you need to set S3Connection.DefaultHost to S3 endpoint with region name: e.g.,
S3Connection.DefaultHost='s3-eu-west-1.amazonaws.com'

@evdoks Thanks for the troubleshooting. A PR and a test would be greatly appreciated.

@evdoks IIRC, in one of the recent PRs (merged already), we added passing parameters from smart_open directly through to S3.
Does that help with passing calling_format here?

calling_format explicitly added in https://github.com/RaRe-Technologies/smart_open/pull/83

@ziky90 We replaced boto with boto3 in the S3 subsystem. Could you please check if the problem with dots in bucket names still occurs?

I close issue because it works fine with bucket name with dots (like com.test-gensim.wow) and current master fbc82cc04bf92e8c588c49c43ff5aff8234ea87d


Add description field

Could we have a description field (or any other name) that could be used to fill the title attribute of a link. This would be a plus for SEO and accessibility.

To be considered in the next version, as for now you're able to extend the navigation item content type using Strapi Extensions: https://docs-v3.strapi.io/developer-docs/latest/development/admin-customization.html#customization-options


[T4A6][W13-B1]Lang Yanbin

Ready for review

@langyanbin0314
Some comments added. Please ack & close the PR after reading comments.


Call protocol methods on private classes

Consider the following case:
@protocol MyProtocol1<NSObject>
- (void)method1;
@end

@protocol MyProtocol2<NSObject>
- (void)method2;
@end

@interface MyClass : NSObject
+ (id<MyProtocol1, MyProtocol2>)getObject;
@end

There should be a non-hacky way to call method1/method2 on the object returned by MyClass.getObject().
This is required by the NativeScript modules for custom page transitions.
ping @hamorphis @PanayotCankov

I'd suggest that we create constructor functions and wire properly the inheritance chain for these private classes at runtime. These runtime created classes should expose the methods from any known public protocols implemented by the private Obj-C class. Then we should be able to wrap the private instances so we can easily call from JavaScript the methods conforming to the protocols, just as we can from Objective-C or Swift.

@PanayotCankov I agree that your solution leads to more naturally calling protocol methods, and instead of writing
MyProtocol1.prototype.method1.call(instance, ...);

you will be able to achieve the same with
instance.method1(...);

However, your solution is not contrary to the currently implemented one. If we take your approach we still have to attach protocol prototypes to their corresponding constructor functions, because IMO its quite natural and also this will make instanceof to work with protocols, too.
However, this PR has value on its own because it allows you to call methods from protocols listed in <...> which was not possible till now (without using some hacky workarounds), and gives us the possibility to make it more elegant in the future.

@PanayotCankov @ivanbuhov We can't rely on Objective-C introspection to provide reliable information on what protocols a class conforms to. The most recent example is UIViewControllerContextTransitioning - this protocol isn't even registered with Objective-C at runtime, so there is no way for us to query whether or not an instance of a private class conforms to it, even though the private class does actually conform to it by implementing all methods from the protocol.


Cannot import RESTClient

I'm trying to get this to run on macOS since, I assume, I am unable to run the docker image until I have populated the various config files.
After installing pyenv in order to install ynam via pip install ynam I get this error when trying to run ynam --quickstart
For Docker, have you considered passing in the required values via environment variables? Or provide documentation on how to manually populate the yaml files?
❯ ynam --quickstart
Traceback (most recent call last):
  File "/Users/hackmodford/.pyenv/versions/3.10.4/bin/ynam", line 5, in <module>
    from ynam.main import main
  File "/Users/hackmodford/.pyenv/versions/3.10.4/lib/python3.10/site-packages/ynam/main.py", line 5, in <module>
    from .mint_api import MintAPI
  File "/Users/hackmodford/.pyenv/versions/3.10.4/lib/python3.10/site-packages/ynam/mint_api.py", line 5, in <module>
    from mintapi import RESTClient, SeleniumBrowser
ImportError: cannot import name 'RESTClient' from 'mintapi' (/Users/hackmodford/.pyenv/versions/3.10.4/lib/python3.10/site-packages/mintapi/__init__.py)

Issue also happened when following instructions in a python docker container, so it's not related to macOS

Happy New Year!

After installing pyenv in order to install ynam via pip install ynam I get this error when trying to run ynam --quickstart
...
ImportError: cannot import name 'RESTClient' from 'mintapi' (/Users/hackmodford/.pyenv/versions/3.10.4/lib/python3.10/site-packages/mintapi/init.py)

Did you install my fork of the mintapi package? The import error you're getting is usually because pypi tries to be helpful and install the canonical version from mintapi/mintapi, which doesn't yet have my changes. (PR pending)

For Docker, have you considered passing in the required values via environment variables? …

I hadn't, actually, mostly because I've never used that aspect of Docker.
I'd welcome a PR, if you have time to devote to that.

… Or provide documentation on how to manually populate the yaml files?

(assuming you mean the json-formated ~/.ynamrc, as this project doesn't use yaml)
Can you elaborate a bit on why this would be needed?

Happy New Year!

After installing pyenv in order to install ynam via pip install ynam I get this error when trying to run ynam --quickstart
...
ImportError: cannot import name 'RESTClient' from 'mintapi' (/Users/hackmodford/.pyenv/versions/3.10.4/lib/python3.10/site-packages/mintapi/init.py)

Did you install my fork of the mintapi package? The import error you're getting is usually because pypi tries to be helpful and install the canonical version from mintapi/mintapi, which doesn't yet have my changes. (PR pending)

For Docker, have you considered passing in the required values via environment variables? …

I hadn't, actually, mostly because I've never used that aspect of Docker. I'd welcome a PR, if you have time to devote to that.

… Or provide documentation on how to manually populate the yaml files?

(assuming you mean the json-formated ~/.ynamrc, as this project doesn't use yaml) Can you elaborate a bit on why this would be needed?

Thank you, that fixed my problem with the REST error, but now I'm running into a new problem. After punching in my YNAB API key and choosing my budget, I get the following error messages:
Traceback (most recent call last):
File "/opt/homebrew/lib/python3.9/site-packages/ynam/ynab_api.py", line 653, in _decode
return answer['data']
KeyError: 'data'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/homebrew/bin/ynam", line 8, in
sys.exit(main())
File "/opt/homebrew/lib/python3.9/site-packages/ynam/main.py", line 20, in main
handleArgs()
File "/opt/homebrew/lib/python3.9/site-packages/ynam/main.py", line 48, in handleArgs
sys.exit(run())
File "/opt/homebrew/lib/python3.9/site-packages/ynam/quickstart.py", line 44, in run
updateStash('ynab_account_id', usersChoice(ynapi.get_accounts())['id'])
File "/opt/homebrew/lib/python3.9/site-packages/ynam/ynab_api.py", line 127, in get_accounts
return _decode(self._get(url))['accounts']
File "/opt/homebrew/lib/python3.9/site-packages/ynam/ynab_api.py", line 655, in _decode
raise Exception(answer['error'])
Exception: {'id': '404.2', 'name': 'resource_not_found', 'detail': 'Resource not found'}

I'm guessing that's the same error as #22 , which got fixed just the other day. pip install -U ynam an drerun

I'm guessing that's the same error as #22 , which got fixed just the other day. pip install -U ynam and rerun

Yep, that fixed my issue, but still running into other issues when running the ynam command. Will go back through the instructions and see what may be going on.

I'm closing this issue as resolved. Feel free to open up another issue with any further problems


Ft username store contract

[ ] issue #issue
[ ] follows contribution guide
[ ] code change includes tests
[ ] breaking change

Starknet smart contract for username store
Caller of the contract can claim a name, the name is link to his address
Caller of the contract can transfer a name, if he/she is the one that claim the name

@b-j-roberts i have fix the changes as requested , kindly review them

@b-j-roberts kindly review the updated test functions


[hardware][cuda] use device id under CUDA_VISIBLE_DEVICES for get_device_capability

fixes https://github.com/vllm-project/vllm/issues/6213

Should we replace all usages of torch.cuda.get_device_capability() with this? For example attention selector and ROCm attention now still use the native torch one, which may result in the same issue.

Should we replace all usages of torch.cuda.get_device_capability() with this? For example attention selector and ROCm attention now still use the native torch one, which may result in the same issue.

I replaced most of them in #6080 . The rest are mostly used under is_hip, and they still use torch.cuda.get_device_capability() . In rocm, this function does not initialize device.

@youkaichao I forget where but I ran into CI issues with current_platform.get_device_capability() hitting an exception on CPU tests because current_platform returns as None in that case. Should we make better dummy behavior for CPU/non-supported platforms, such as returning 0?

I forget where but I ran into CI issues with current_platform.get_device_capability() hitting an exception on CPU tests because current_platform returns as None in that case.

links would be appreciated. I don't think cpu code should call this function. even before this change, torch.cuda.get_device_capability() would also error out in cpu case.

Here are some examples

https://github.com/vllm-project/vllm/blob/main/vllm/attention/selector.py#L151
https://github.com/vllm-project/vllm/blob/main/vllm/attention/backends/rocm_flash_attn.py#L270
https://github.com/vllm-project/vllm/blob/main/vllm/platforms/rocm.py#L15

How about this one?
https://github.com/vllm-project/vllm/blob/main/vllm/attention/selector.py#L151

@comaniac these should be called after process initialize cuda, so they are fine. we can also replace them, though.

@comaniac these should be called after process initialize cuda, so they are fine. we can also replace them, though.

Yeah it would be safer to replace this one as well.

Merge first. I'll change the use in selector in another PR.


Support Multiple Inheritance

Problem
I have the following config.mjs:
export default {
  optionalFeatures: ["crossFileRelations"],
  baseSchema: BaseSchema,
  includeFiles: [...globModels(BaseSchema, "prisma/models/**/*.prisma")],
  extended: {
    "user.prisma:User": "vendor.prisma:Vendor"
  },
}

I would like to be able to extend multiple models, as such:
  extended: {
    "user.prisma:User": ["vendor.prisma:Vendor", "vendor.prisma:Customer", ...]
  },

This is currently not possible.
Suggested solution
The extended config parameter should support a string or list of strings as the type, such that a model can extend multiple other models.
Alternatives
Honestly, the current DX for defining cross-file relations and extensions both are quite bad. This package should move long-term to support the following syntax in .prisma files:
# import './Project.prisma'

model User extends Vendor, Customer {
   ....

  projects          Project[]
}

The import statement can be completed by plugging in the graphql-import NPM package or a similar alternative.
Additional context
This package is much needed, great job building it. There are multiple issues open on Prisma's Github for multiple years that this package resolves!

Hello @notadamking!
Thank you for this suggestion. I'm happy to tell you that this will be implemented in my Rust rewrite of Prisma Util, Turboprisma. I plan on deprecating Prisma Util when it has been fully replaced by Turboprisma, complete with a VSCode extension for auto-completion.
This is the syntax I had planned:
abstract model Dates {
    createdAt DateTime @default(now())
    updatedAt DateTime @updatedAt
}

model Test extends Dates {
    id Int @id @default(autoincrement())
}

// Results in:
model Test {
    id Int @id @default(autoincrement())
    createdAt DateTime @default(now())
    updatedAt DateTime @updatedAt
}

Turboprisma

Amazing! Please update this thread as the feature becomes available in Turboprisma, I will use it immediately.
In addition, I really think supporting cross-file imports, as opposed to defining relations in the config and requiring duplicate shims for each relation in each model file, would go a LONG way to improve the DX.

Of course, that is also planned, including the ability to define custom attribute and validators plus real-time database listeners and refined types. I'm also getting ready for the next release, which will bring the migration engine with fully type-safe data migrations written in TypeScript.

Of course, that is also planned, including the ability to define custom attribute and validators plus real-time database listeners and refined types. I'm also getting ready for the next release, which will bring the migration engine with fully type-safe data migrations written in TypeScript.

This is great to hear. Any chances there is a test build I could play around with?
Otherwise, if a turboprisma build is not on the near horizon, would it be simple to make the change as first described (support a string of list extensions) to this package? I have not dived deep into the codebase yet to see how difficult this would be.

Well, the migration part should be released in about ~4-5 days. After that, I estimate about a week or so for developing model inheritance. I think that it would be better to wait a bit for the Turboprisma release since it will be much better for the DX.
But even then, if it takes more than a week, I will make the change in this repository and notify you. How does that sound?

Test builds will also be available once the new Turboprisma Version Manager is released, so I can enable telemetry for those releases for feedback purposes.

Well, the migration part should be released in about ~4-5 days. After that, I estimate about a week or so for developing model inheritance. I think that it would be better to wait a bit for the Turboprisma release since it will be much better for the DX.
But even then, if it takes more than a week, I will make the change in this repository and notify you. How does that sound?

Perfect, you're awesome!

Well, the migration part should be released in about ~4-5 days. After that, I estimate about a week or so for developing model inheritance. I think that it would be better to wait a bit for the Turboprisma release since it will be much better for the DX.
But even then, if it takes more than a week, I will make the change in this repository and notify you. How does that sound?

bump :)

Hi @notadamking!
I apologize for the delay. We've had some issues with the type generation for Data Migrations and had to rewrite our implementation. But now it works and is 80% done.

Hi @notadamking!
I apologize for the delay. We've had some issues with the type generation for Data Migrations and had to rewrite our implementation. But now it works and is 80% done.

Great to hear! Do you have an updated ETA? Still very eager to use this.
I am currently using Zenstack, after testing pretty much every similar package I could find. It works good, but doesn't fully solve inheritance (abstract models can't define relations, extending models can't override optional parameters to be required, models can't use an abstract model as a field in another model, nested inheritance is buggy, etc.)
Though one thing that Zenstack solves, which would be great to see in turboprisma, is access control. Unfortunately Zenstack's access control doesn't support roles and permissions very well, and doesn't doesn't support field-level access control (i.e. only letting a user see their own phone number, while users in their organization can see their name/email).

Hi again @notadamking!
The model inheritance issues you've mentioned will definitely not be a problem in Turboprisma. Due to us interfacing with the schema at engine level (something that Zenstack doesn't do), we can offer more granular control. Our planned feature set for model inheritance includes the ability to override any field definition, have nested inheritance with ease just like you would in something like Java and the ability to use inheritance at database level for Postgres.
As for the relation part on abstract models, could you provide an example of how that would look like?

How would this be represented in the database?

How would this be represented in the database?

I think the simplest way to represent this in a database would be to add the inherited class as a foreign key to the table of the extending model, then at runtime use joins to present the data from the database in the format expected in the schema.
CREATE TABLE project_owner (
  id SERIAL PRIMARY KEY,
  extends_vendor_id INTEGER REFERENCES extends_vendor(id),
  extends_customer_id INTEGER REFERENCES extends_customer(id),
  ... -- other project_owner fields
);

CREATE TABLE user (
  id SERIAL PRIMARY KEY,
  extends_project_owner_id INTEGER REFERENCES extends_project_owner(id),
  ... -- other user fields
);

where in this example, User extends ProjectOwner, which extends Vendor and Customer.

PostgreSQL also supports a form of table inheritance natively, but I'm not entirely sure it works the same as the above description using SQL Join statements and separate tables.
https://www.linuxtopia.org/online_books/database_guides/Practical_PostgreSQL_database/PostgreSQL_x13546_002.htm

How would this be represented in the database?

I think the simplest way to represent this in a database would be to add the inherited class as a foreign key to the table of the extending model, then at runtime use joins to present the data from the database in the format expected in the schema.
CREATE TABLE project_owner (
  id SERIAL PRIMARY KEY,
  extends_vendor_id INTEGER REFERENCES extends_vendor(id),
  extends_customer_id INTEGER REFERENCES extends_customer(id),
  ... -- other project_owner fields
);

CREATE TABLE user (
  id SERIAL PRIMARY KEY,
  extends_project_owner_id INTEGER REFERENCES extends_project_owner(id),
  ... -- other user fields
);

where in this example, User extends ProjectOwner, which extends Vendor and Customer.

The problem is, if the referenced field has an abstract model as the type, it would be a bit harder to work with, leading to some other questions like what model should take precedence? Should it accept any model that inherits from that abstract model? Should it only accept one table for the relation?

How would this be represented in the database?

I think the simplest way to represent this in a database would be to add the inherited class as a foreign key to the table of the extending model, then at runtime use joins to present the data from the database in the format expected in the schema.
CREATE TABLE project_owner (
  id SERIAL PRIMARY KEY,
  extends_vendor_id INTEGER REFERENCES extends_vendor(id),
  extends_customer_id INTEGER REFERENCES extends_customer(id),
  ... -- other project_owner fields
);

CREATE TABLE user (
  id SERIAL PRIMARY KEY,
  extends_project_owner_id INTEGER REFERENCES extends_project_owner(id),
  ... -- other user fields
);

where in this example, User extends ProjectOwner, which extends Vendor and Customer.

The problem is, if the referenced field has an abstract model as the type, it would be a bit harder to work with, leading to some other questions like what model should take precedence? Should it accept any model that inherits from that abstract model? Should it only accept one table for the relation?

I believe the following schema accurately represents what you are referring to:
abstract model ProjectOwner extends Vendor, Customer {
   ...
}

model User extends ProjectOwner {
  ...
}

model Organization extends ProjectOwner {
   ...
}

model Project {
   ...

   owner:         ProjectOwner
}

which would be represented in the database as follows:
CREATE TYPE project_owner_type AS ENUM ('user', 'organization');
CREATE TYPE user_type AS ENUM ('user');
CREATE TYPE organization_type AS ENUM ('organization');
CREATE TYPE project_type AS ENUM ('project');

CREATE TABLE project_owner (
  id SERIAL PRIMARY KEY,
  type project_owner_type,
  extends_vendor_id INTEGER REFERENCES extends_vendor(id),
  extends_customer_id INTEGER REFERENCES extends_customer(id),
  ... -- other project_owner fields
);

CREATE TABLE user (
  id SERIAL PRIMARY KEY,
  type user_type,
  extends_project_owner_id INTEGER REFERENCES extends_project_owner(id),
  ... -- other user fields
);

CREATE TABLE organization (
  id SERIAL PRIMARY KEY,
  type organization_type,
  extends_project_owner_id INTEGER REFERENCES extends_project_owner(id),
  ... -- other user fields
);

CREATE TABLE project (
  id SERIAL PRIMARY KEY,
  type project_type,
  project_owner_id INTEGER REFERENCES project_owner(id),
  ... -- other user fields
);

In which case I believe the simplest solution is to store abstract relationships as simply a foreign key id (as above) referring to the top-level class (i.e. a User or Organization), and do the interpolation back to full class instance with joins. This would also require adding a type field to every table (or at least every abstract table), such that if the model is later extended, the extended type can be recovered.
Then the answers to your questions becomes trivial:

Which model should take precedence? The top level model should always be restored, the defined model in the schema should always be the foreign relationship stored in the database.

Should it accept any model that inherits from that abstract model? Yes, because you would only store the reference to the abstract class in the database field, then restore back to the full class at runtime using joins using the type field on the abstract model.

Should it only accept one table for the relation? Yes. Both in the Prisma schema and in the database, the relation key would be referring to the abstract (or extended) model.

Then I know you may be thinking, what about nested relations? In the above model, they just work. For example, if we added a new model in the schema:
model Admin extends User {
   ...
}

then we would simply update the database with the following:
CREATE TYPE user_type AS ENUM ('user', 'admin');
CREATE TYPE admin_type AS ENUM ('admin');

CREATE TABLE admin (
  id SERIAL PRIMARY KEY,
  type admin_type,
  extends_user_id INTEGER REFERENCES user(id),
  ... -- other user fields
);

And so long as the join statements or model interpolation is correct, everything should be set up as expected and follow traditional object orientation, I believe.


Inventory problem

Hello, when you force the closure of the inventory (For example when we save a recipe), we must disconnect and reconnect to be able to reopen our inventory or another interface

This is a rare error come from some minecraft client...
Try InvCrashFix


Unable to do perform RTL of Fixed wing in sitl

when RTL is given to Fixed wing it returns back to launch  position as my parameter RTL_LAND_DELAY  is  60 it should loiter at home position for 60 sec and land but when it starts descending altitude and all of sudden again gains altitude  go faraway please help me in this @JonasVautherin

@airsimdevdd this sounds like a PX4 issue and not a MAVSDK issue, but it's hard to say without log. Can you please upload a logfile of when this happens to logs.px4.io and then link it here?


add stopProcess

Dependent on https://github.com/xdblab/xdb/pull/42

@longquanzheng This MR is ready now.


cyber: unstable-2023-08-11 -> unstable-2023-08-24

Diff: https://github.com/fubark/cyber/compare/242ba2573cbac2acecc8c06878a8d754dd7a8716...be76bc13590285cffa502c3c97470a80ff1f27bd
Description of changes

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

[ ] For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[x] Tested basic functionality of all binary files (usually in ./result/bin/)
23.11 Release Notes (or backporting 23.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Result of nixpkgs-review pr 251627 run on x86_64-linux 1

1 package built:

cyber


Don't seed global RNG

This line caused a tricky intermittent test failure for us by creating frequent collisions of random numbers. Plus, it is just a good practice in general to avoid seeding the global RNG in a library function.
Example:
func TestA(t *testing.T) {
  // init canal (seeds rng)
  // some test code that doesn't call rand
}

func TestB(t *testing.T) {
  id1 := rand.Intn()
  // init canal (seeds rng to original seed if <1s has elapsed)
  id2 := rand.Intn()
  // id1 == id2
  // run an update id1 -> id2 in db, canal does not receive an update event, test fails
}

@siddontang can you have a look at this plz?

Thanks @austintaylor


fix:chown socket file to belong to nobody

In order to solve this bug, the socket created by apisix/runner/server/server.py now belongs to nobody so apisix runners could write on it.
I choose not to follow go plugin's chmod approach here, for it is to permissive / dangerous and could lead to privilege escalation

@spacewander , @shuaijinchao , could you review this MR please?

@spacewander @shuaijinchao Plz give me a approval :)


Added the Old Salli voice

@narutofan420 please close the pull request

This looks untested; Has this been seen working?

Okay then.

Sorry for reopening, misclick.


[9.4.x] ISPN-10666 Eviction get stuck with  configuration and optim…

…istic transactions
https://issues.jboss.org/browse/ISPN-10666

integrated! thanks @wburns


Minimal support for fn types in extern statics

This is enough to support the usage of static functions in jemalloc-sys. It might however break on more complicated function types (functions taking and returning functions as arguments).

tidied it up, and covered some corner cases involving function pointers taking/returning other function pointers as arguments


Various fixes from upstream:

properly handle sys packages when cross building
add --conda-channel to requirements.txt type of files
fix issue when locally building a package that already had a local identifier
better handling of determining if a newer package needs to be built when pointing to a local directory or file
add support to specify additional packages for builder environments thereby supporting authentication using GOOGLE_APPLICATION_CREDENTIALS
improved support for python 3.7
more efficient environment building when building an environment that just adds ipykernel
fix issue with micromamba in some cases (it would "hang" when compiling .pyc files)

I confirm this solved the use case of GOOGLE_APPLICATION_CREDENTIALS, thanks @romain-intel


Create FNIC Concept

Draft PR to document FNIC

got it! thanks Kristina for capturing the essence! will create a more detailed doc on this one.


Complex owned types slow down query by creating unnecessary self joins

I have tables with multiple complex owned types but for some reason when i do a simple select it will make a self join for each complex type within the table.
I.e.
.Select(x => new OrderView
{
    Guid = x.TransportOrderGuid,
    Timestamp = x.Timestamp,
    TransportOrderNumber = x.TransportOrderSharedData.TransportOrderNumber,
    ExternalTransportOrderNumber = x.TransportOrderData.ExternalTransportOrderNumber,
    DeliveryNoteNumber = x.DeliveryNoteData.DeliveryNoteNumber,
    ExternalDeliveryNoteNumber = x.DeliveryNoteData.ExternalDeliveryNoteNumber,
    //and so on
}

Is there a way to force disable self joins for complex types? This is really slowing down the whole thing. The reason why I'm using complex types was because I wanted to reduce joins to improve perfomance.
So instead of having a separate Table with Addresses (containing a address type enum to differentiate the different addresses) I now have a complex Address type:
public TransportAddress Customer {get;set;}
public TransportAddress Supplier {get;set;}

instead of
public List<TransportAddress> Addresses {get;set;}

But now instead of multiple joins to flatten out different addresses it's not creating unnecessary joins for owned types...
Further technical details
EF Core version: 3.1 preview 3
Database provider: Npgsql.EntityFrameworkCore.PostgresServer
Target framework: .NET Core 3.1
Operating system: Win 10
IDE: Visual Studio 2019 Preview (latest)
EDIT:
My guess is this is done to see if the complex type is null ==> Is there a way to maybe make a complex type required so this check isn't necessary?
I would prefer to disable the null check for complex types for a query somehow...  => returning the default value for required fields and speeding up the query

Duplicate of https://github.com/aspnet/EntityFrameworkCore/issues/12100 and https://github.com/aspnet/EntityFrameworkCore/issues/18869


[Bug] fe-core: Compilation failure

Search before asking

[X] I had searched in the issues and found no similar issues.

Version
master    8a6c9d8   (#11207)
What's Wrong?
fe-core compile faild
What You Expected?
compile succeed
How to Reproduce?
Fix the error by commenting this line of code
doris/fe/fe-core/src/test/java/org/apache/doris/utframe/TestWithFeService.java:111
//Catalog.getCurrentCatalog().clear();
Anything Else?
No response
Are you willing to submit PR?

[X] Yes I am willing to submit a PR!

Code of Conduct

[X] I agree to follow this project's Code of Conduct

fixed on #11215


Instruct robots to ignore proxy

Robots are already crawling my takahē instance, which is fine, but crawling the proxy is a good way to burn bandwidth. Also Mastodon appears to have a similar block by default (e.g. https://mastodon.social/robots.txt)

Good thinking!


Eliminate static state in extension features

Eliminate static state in TraceLoadedDotNetRuntimeExtensions
Eliminate static state in TraceProcessesExtensions

Codecov Report

Merging #261 into master will increase coverage by <.01%.
The diff coverage is 27.58%.

@@            Coverage Diff             @@
##           master     #261      +/-   ##
==========================================
+ Coverage   24.32%   24.32%   +<.01%
==========================================
  Files         197      197
  Lines      112379   112389      +10
  Branches    11208    11204       -4
==========================================
+ Hits        27331    27334       +3
- Misses      85021    85028       +7
  Partials       27       27

Flag
Coverage Δ

#2015
24.31% <27.58%> (ø)
:arrow_up:

#2017
24.32% <27.58%> (ø)
:arrow_up:

#Debug
15.78% <0%> (-0.01%)
:arrow_down:

#Release
14.85% <0%> (-0.01%)
:arrow_down:

Impacted Files
Coverage Δ

src/TraceEvent/Computers/TraceManagedProcess.cs
14.12% <27.27%> (+0.15%)
:arrow_up:

src/TraceEvent/Computers/TraceProcess.cs
11.11% <28.57%> (-0.4%)
:arrow_down:

src/PerfView/StackViewer/StackWindow.xaml.cs
34.39% <0%> (-0.04%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update da21a66...a282726. Read the comment docs.

Codecov Report

Merging #261 into master will increase coverage by <.01%.
The diff coverage is 27.58%.

@@            Coverage Diff             @@
##           master     #261      +/-   ##
==========================================
+ Coverage   24.32%   24.32%   +<.01%
==========================================
  Files         197      197
  Lines      112379   112389      +10
  Branches    11208    11204       -4
==========================================
+ Hits        27331    27334       +3
- Misses      85021    85028       +7
  Partials       27       27

Flag
Coverage Δ

#2015
24.31% <27.58%> (ø)
:arrow_up:

#2017
24.32% <27.58%> (ø)
:arrow_up:

#Debug
15.78% <0%> (-0.01%)
:arrow_down:

#Release
14.85% <0%> (-0.01%)
:arrow_down:

Impacted Files
Coverage Δ

src/TraceEvent/Computers/TraceManagedProcess.cs
14.12% <27.27%> (+0.15%)
:arrow_up:

src/TraceEvent/Computers/TraceProcess.cs
11.11% <28.57%> (-0.4%)
:arrow_down:

src/PerfView/StackViewer/StackWindow.xaml.cs
34.39% <0%> (-0.04%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update da21a66...a282726. Read the comment docs.

@jeffschwMSFT

@sharwell I added a few comments - they hit similar scenarios, reuse and multiple calls on the same source.  What sort of validation suite are you using for these?  Many of the harder issues with TraceEvent come up when you drill into source reuse, concurrency, and live events.  All these scenarios are not just possible they are hit in our live etl processing.

@jeffschwMSFT I don't currently have tests for this. If you can describe a test case in a fair amount of specificity, I can get it coded as a unit test.
These changes are focused on eliminating potential negative interactions under concurrent use of independent objects. I was not looking at source reuse. In the latter case, this pull request only has one substantial impact, which is the elimination of clearing the accumulated data under a second call to Need* for the same source. If this is a scenario you need, I can easily add it back without losing the benefits this pull request offers for concurrent usage scenarios.

@sharwell I am having a hard time understanding the motivation and underlying value of these changes.  Can you help me understand?  The net-net of the change is removal of one static to track source - which frankly was a loose guarantee / hack.  The other changes are just moving things around.

I am having a hard time understanding the motivation and underlying value of these changes. Can you help me understand?

My general motivation comes on two fronts:

Ensure that two independent well-behaved¹ users of the TraceEvent library (i.e. users operating on distinct objects and distinct input files) do not interfere with each other in a concurrent application
Where reasonable, protect against mutations to static state which is not supposed to change.

When applied, these practices improve the reliability of the library in distribution and also help our own development through details like preventing one unit test from affecting the behavior of a following unit test.
¹ Well-behaved means adhering to the requirements documented for various methods in the library with respect to exposed static state that is technically mutable but is expected to not actually be mutated by library users.

@sharwell What case is failing today with the current implementation?  In my mind the well-behaved case is load an ETL, process it, then shutdown the process.  We do this regularly and much more.  A great example of the well behaved case is Perfview and I am not aware of a failure/issue there.

@jeffschwMSFT The failing case today is two independent threads concurrently processing two independent trace files. The last one to call NeedProcesses will result in the first, on an unrelated processing thread, having assertion failures.

We do that all the time - this is a common case.  What is the result when this happens?  I am seeing it succeed.  Related, what portion of this change materially makes it better?

We do that all the time - this is a common case. What is the result when this happens? I am seeing it succeed. Related, what portion of this change materially makes it better?

The most obvious problematic cases only occur in debug builds of the TraceEvent library (i.e. you actually execute your code against a debug build of TraceEvent, as opposed to the more common action of executing a debug build of a tool which references the TraceEvent library). For release builds, the only observable impact in TraceProcesses would be extra-accumulation of the amount of time spent in a process here after using NeedLoadedDotNetRuntimes or NeedProcesses a second time to reset accumulated state.
For duplicate callback registration in NeedLoadedDotNetRuntimes, the resulting aggregation errors are a bit more difficult to be certain of without more work, but I couldn't find anything to indicate that you would experience anything more severe than (presumably subtle) inaccuracy of results if this race condition is hit.

Debug TraceEvent has a number of issues - it is noble and much needed to clean those up.  Though the asserts I have seen are far deeper within the setup callbacks and not at this level.
I agree the following scenario is problematic (a good bug find): using(source) { source.NeedProcesses(); source.Process(); source.NeedProcesses(); source.Process(); }.  Though the fix for that is different than the proposed.  I would recommend tackling that issue within the context of also keeping the right semantic for the following, which is far more common: using(source) { source.NeedProcesses(); source.NeedProcesses(); source.Process(); }
We need to find a way to fix the issue you encountered and keep the existing compatibility.

I wrote some initial tests and should have a new commit ready in the morning with them. 👍

To start please merge the relevant commits, after that I will then take a deeper look.
The semantics for NeedProcesses and NeedLoadedDotNetRuntimes should be as follows:

On first call - setup callbacks and create state
On subsequent calls - do nothing
After processing and calling - clear state, but do not re-establish callbacks

As for scenarios to test I would do the following:

PerfView case - using(sourceA) { sourceA.NeedsProcesses(); sourceA.NeedsLoadedDotNetRuntimes(); sourceA.Process(); }
for(many different sources) { using(source) { source.NeedsProcesses(); source.NeedsLoadedDotNetRuntimes(); source.Process(); } }
using(sourceA) { sourceA.NeedsProcesses(); sourceA.NeedsLoadedDotNetRuntimes(); sourceA.Process(); } /* the same for sourceB / / then sourceA again */
etc. the cases in the tests seem to cover interesting scenarios - the ones I am most concerned with are thread concurrent and multiple different ETLs with occassional duplication

Some of the test cases are really non-scenarios.  What is the customer use case for doing sourceA.Process() multiple times in a row?  This feels like a debug scenario.

Some of the test cases are really non-scenarios. What is the customer use case for doing sourceA.Process() multiple times in a row?

The test doesn't actually call sourceA.Process() two times in a row. The test is intended to deterministically reproduce a concurrency bug by forcing what would normally be two independent operations on two different sources to execute in a specific order. It is a specific linearization of two independent operations.

To start please merge the relevant commits, after that I will then take a deeper look.

@vancem Can you take a look at merging #274 and then #255 to help out with this?
@jeffschwMSFT Prior to that, I would suggest you focus exclusively on reviewing the tests in TraceEventDispatcherExtensionsTests.cs. They are unique to this pull request and are intended to be valid test cases regardless what, if any, other code changes are made.

The semantics for NeedProcesses and NeedLoadedDotNetRuntimes should be as follows:

On first call - setup callbacks and create state
On subsequent calls - do nothing
After processing and calling - clear state, but do not re-establish callbacks

Currently we are close:

On first call: setup callbacks and create state ✔
On subsequent calls: do not setup callbacks ✔  but reestablishes state, which should be idempotent ❓
After processing and calling: do not setup callbacks ✔ and reestablishes state ✔ but I was not able to create a test that proved it

Note that the following test failed when I attempted it, which suggests something is not getting cleared properly:
source.NeedProcesses();
source.Process();

source.NeedProcesses();
source.Process();

processTimes = source.Processes().ToDictionary(p => p.ProcessID, p => p.CPUMSec);

However, the equivalent test for NeedDotNetLoadedRuntimes was working correctly, so the state reset failure appears to be limited to TraceProcesses. I filed #276 for this issue.

As for scenarios to test I would do the following:

Most of these were covered already, and I added additional coverage in 86b70e5df5b9706028c57f20ecbdff45b3967745.

@sharwell merge as in merge the commits, not merge with master.  Please collapse the changes so I can see the true deltas from the current source.  What realworld scenarios are represented in the test cases?

@jeffschwMSFT This has now been updated.

@sharwell as discussed please merge relevant commits and then I will take a deeper look.

@jeffschwMSFT I rebased and merged to have three distinct units of work:

Add tests which demonstrate the observable problems and known use cases
Self-contained corrections to TraceProcessesExtensions
Self-contained corrections to TraceLoadedDotNetRuntimeExtensions

This change looks good.  Thanks for adding tests.

Updated after repository rewrite


When will you provide the code for your implementation?

Yes, I will provide the five public  FSD datasets along with my own MS-FSDB, and I’ll also include the data processing code and evaluation methods. However, I’m currently a bit busy with paper writing these days, so it might take some time. Once I’ve uploaded the dataset and code, I’ll notify you as soon as possible.
If possible, I'll take the time to submit these datasets and code, this weekend.


Floating action button not closing!

I updated flutter to version 3.24.0
I updated the plugin to version 2.2.0
And whenever I open the floating action button I can't close it using the close button, it just doesn't work.

I am also seeing this issue with flutter 3.22.0, plugin 2.3.0.  The close button doesn't always work but the action buttons do. Also it appears that state of the extended fab is always open. Issue worse on iOS but can be replicated on Android.
[✓] Flutter (Channel stable, 3.22.0, on macOS 15.0.1 24A348 darwin-arm64, locale en-US)
• Flutter version 3.22.0 on channel stable at /Users/scottmusson/fvm/versions/3.22.0
• Upstream repository https://github.com/flutter/flutter.git
• Framework revision 5dcb86f68f (6 months ago), 2024-05-09 07:39:20 -0500
• Engine revision f6344b75dc
• Dart version 3.4.0
• DevTools version 2.34.3


Invoke does not work on windows

run does not work on windows - it's required specific config file and even with it will fail due to virtualenv not active. Consider:

Switch invoke to another library
Use vagrant or docker
Drop windows support at all

Invoke was dropped from the project. No more fancy tasks.


Added new API to enable disable branch SDK tracking

Added new API to enable disable branch SDK tracking

@hardikhexa can you provide a super short code snippet for how a developer should use this method? Or can you link to the exact line in the testbed app you've built?

@hardikhexa can you provide a super short code snippet for how a developer should use this method? Or can you link to the exact line in the testbed app you've built?

Is this code snippet correct, or did I get the syntax wrong?
m.branchSdkObj.disableTracking(true)

Is this code snippet correct, or did I get the syntax wrong?
m.branchSdkObj.disableTracking(true)